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Introduction 


Purpose of the Self-Study Guide 


The Every Student Succeeds Act (ESSA) challenges state education agencies (SEAs) to improve student 
outcomes by addressing the student-, teacher-, and school-level factors that drive achievement gains. 


This Guide for Identifying Evidence-Based Interventions for School Improvement is intended to help 
State Education Agencies (SEAs) carefully consider the evidence supporting intervention options that 
they will require or recommend in their state ESSA plan and funding applications. SEAs could indicate 
in their state ESSA plan how they have used or will use the self-study process to identify interventions, 
in partnership with stakeholders. The purpose of the guide is to help SEAs: 


1. evaluate the evidence base for interventions as they identify those to be included in the state 
plan for ESSA as options for schools in need of comprehensive or targeted support, 


2. determine the interventions that have strong evidence, and are relevant and appropriate to 
meeting the needs of the Local Education Agencies (LEAs), and 


3. plan to provide resources for LEAs to help them choose the best evidence-based option(s) for 
schools in need of comprehensive or targeted support to include in school improvement plans. 


Flexibility with Responsibility — The Roles of the SEA and LEA 


SEAs and LEAs are charged with implementing ESSA, with states being asked to ensure that LEAs are 
implementing evidence-based strategies, activities, and interventions in schools in need of significant 
improvement. Throughout this document, evidence-based strategies, activities, and interventions will 
be collectively referenced as “interventions.” 


In the past, school improvement interventions that LEAs were allowed to use were very prescriptive, 
but ESSA provides states with the flexibility to delineate interventions, or help LEAs select inter- 
ventions, provided they are evidence-based. This flexibility provides LEAs with an opportunity to 
help schools develop improvement plans that may prove to be more effective in increasing student 
achievement. 


ESSA requires that SEAs identify schools in need of comprehensive support and targeted support as 
delineated in this chart provided by the U.S. Department of Education: 


Types of 
Schools 


Lowest- 
Performing 


Low High 
School 
Graduation 
Rate 


Chronically 
Low- 
Performing 
Subgroup 


Types of 
Schools 


Consistently 
Underperform- 
ing Subgroup 


Low-Perform- 
ing Subgroup 


Category: Comprehensive Support and Improvement 


Description 


Lowest-performing five percent of schools in the State 
participating in Title |. 


Any public high school in the State with a four-year 
adjusted cohort graduation rate at or below 67 percent, or 
below a higher percentage selected by the State, over no 
more than three years. 


Any school participating in Title | that was identified 
for targeted support and improvement because it had 
a subgroup of students performing at or below the 
performance of all students in the lowest-performing 
schools and did not improve after implementing a 
targeted support and improvement plan over a State- 
determined number of years. 


Timeline for 
Identification 


At least once 
every three 
years 


At least once 
every three 
years 


At least once 
every three 
years 


Category: Targeted Support and Improvement 


Description 


Any school with one or more consistently underperforming 
subgroups. 


Any school in which one or more subgroups of students is 
performing at or below the performance of all students in 
the lowest-performing schools. These schools must receive 
additional targeted support under the law. 


If this type of school is a Title | school that does not improve 
after implementing a targeted support and improvement 
plan over a State-determined number of years, it becomes a 
school that has a chronically low-performing subgroup and 
is identified for comprehensive support and improvement. 


Timeline for 
Identification 


Annually 


At least once 
every three 
years 


Initial year of 
identification 


2018-2019 


2018-2019 


State- 
determined 


Initial year of 
identification 


2019-2020 


2018-2019 


Along with the flexibility of ESSA comes the responsibility for LEAs, and ultimately SEAs, to ensure that 
evidence-based interventions are selected and implemented so that students attending schools in 
need of comprehensive or targeted support have the best opportunity to improve achievement. LEAs 
and schools in need of comprehensive or targeted support will develop school improvement plans 
which reflect these evidence-based interventions. LEAs will review and approve targeted support 
plans, and SEAs and LEAs will review and approve comprehensive support plans. LEAs must conduct 
a needs assessment for schools identified in need of comprehensive support. Potential interven- 
tions should be evaluated on the basis of school needs and the evidence-based interventions 
selected for implementation should meet the needs of the school. 


The Self-Study Process 


Self-study is a process that facilitates thoughtful investigation and discussion of an issue or topic so 
that decisions can be made through the collaboration of a variety of stakeholders. Although a time 
investment is required to prepare for discussions that focus on the topic or issue, engage in the dis- 
cussions themselves, and subsequently plan for implementation of decisions made by the self-study 
team, the results of this collaboration can be invaluable. 


In order to engage in the self-study process, a team must first be established. The self-study team may 
include school improvement specialists, content area specialists, exceptional student education (ESE) 
and English learner (EL) specialists, as well as those involved in professional development and lead- 
ership at the SEA knowledgeable in school improvement. The SEA may also choose to include repre- 
sentatives from LEAs such as district leaders, teachers and principals to help increase relevance and 
buy-in. A facilitator will then be selected to organize the work and may be chosen by SEA leadership or 
the team itself. Since the goal of this specific self-study is to decide upon interventions that might be 
recommended for schools needing improvement, the facilitator should have deep content knowledge 
of school improvement, be well-organized, a good listener, and be able to lead a discussion that en- 
courages participation from all team members. SEAs may wish to consider using an external facilitator 
such as an individual from a university, Regional Education Laboratory, or Comprehensive Center. 


The self-study process will help SEAs identify the strongest evidence-based interventions that the SEA 
will require or recommend for inclusion in LEA school improvement plans or LEA funding applications. 
SEAs could indicate in their state plan how they have used or will use this self-study to identify inter- 
ventions, in partnership with stakeholders. 


Prior to engaging in this process, the SEA should conduct a needs assessment to clearly identify 

the problems to be addressed (see Figure 2 below). Undoubtedly, schools in need of improvement 
throughout the state will exhibit a variety of issues and problems. It is important that the recommend- 
ed interventions not only have a strong evidence-base, but that they address the issues that schools 
are facing. Figure 1 outlines the general steps in conducting the self-study. 


Figure 1. The Self-Study Process: Conducting the Self-Study 
—9 ; 
= Step 1: Preparation 


Present Overview Team reviews guide and asks questions 
& Review Guide before proceeding to ratings 


Collecting and Facilitator distributes completed 
Evaluating Research SEA Scoring Templates to team 
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provided in the SEA Scoring Guide 


Individual 
Rating 


ins Step 2: Discussion 


Consensus Record recommendation of 
Rating intervention as agreed upon by the team 


Step 3: Planning 


Documenting Complete a detailed plan for next steps 
Next Steps based on urgency, feasibility 


Step 1 is preparation. During this step the facilitator will describe the process to the team and ensure 
that everyone has the same understanding of the work. Each team member will review the sections of 
the self-study guide addressing the collection and evaluation of research and the ESSA Levels of Evi- 
dence, identify one or more potential interventions, and evaluate the level of evidence for them. These 
interventions may fall into the areas that have been identified in the SEA Scoring Guide (described in 
the upcoming Self-Study Guide Tools section), or they may fall into an entirely different category alto- 
gether. This is a critical activity since this guide is unable to address all of the potential interventions a 
state might consider, and more ideas for consideration will improve the results of the discussion step. 
In addition, the team members will complete the SEA Scoring Guide, considering the strategies and 
interventions provided, and reflecting upon whether or not they should be recommended for use in 
LEAs and schools. 


During Step 2 team members discuss all of the various ideas for interventions that the SEA might 
permit or recommend, and the individual ratings that team members assigned on the SEA Scoring 
Template (described in the upcoming Self-Study Guide Tools section) and the SEA Scoring Guide. It is 
during this step that the SEA will settle on the options that LEAs will be authorized to use if the state is 
providing a list of interventions from which LEAs must choose. Having a broad range of strategies and 
interventions is important, but it is equally important that they be based on the best available evi- 
dence. In addition, it is critical that strategies and interventions meet the needs that have been identi- 
fied in the state. 


During the final step, the SEA team members discuss priorities, potential resource development, and 
anticipated challenges in implementation of the strategies. Next steps may be determined with a 
timeline established and team members assigned to tasks. The facilitator leads the discussion and 
information is recorded on the SEA Planning Form (described in the upcoming Self-Study Guide Tools 
section). 


Context for Use of the Self-Study Guide 


Guidance released by the U.S. Department of Education on September 16, 2016 and available at http:// 
www2.ed.gov/policy/elsec/leg/essa/quidanceuseseinvestment.pdf provides a series of steps that can 
promote continuous improvement and support better outcomes for students. These steps include: 


Figure 2. Steps to Promote Continuous Improvement 
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An SEA Guide 
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3. 


Plan for 
Intervention 
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Implement 


Source: Adapted from U.S. Department of Education, 2016. 
The use of this self-study guide will be most helpful in addressing steps two and three above. 


SEAs should select, or help LEAs select, evidence-based interventions (step two) that best meet the 
needs identified in the school-level needs assessment and that address the root causes of underper- 
formance. While the level of evidence should be as strong as possible, it is just as important that the 
interventions meet the needs identified in step one. In addition, the guidance encourages SEAs and 
LEAs to look at the overall body of relevant evidence rather than just one study when selecting inter- 
ventions. Moreover, the evidence base should reflect a preponderance of statistically significant, pos- 
itive effects rather than statistically significant, negative effects. Finally, in cases of minimal evidence, 
the role of strong theory and logic is paramount. 


The guiding questions included in the self-study guide may help team participants consider whether 
an intervention may meet the needs of schools in the state and begin planning for implementation 
(step three). The questions may also provoke thinking about resources available as well as technical 
assistance and support that SEAs may need to offer to LEAs for successful implementation. 


SEA Self-Study Guide Tools 


The SEA Guide for Identifying Evidence-Based Interventions for School Improvement consists of the 
following nine tools: SEA Self-Study Guide Checklist, SEA Facilitator’s Checklist, SEA Team Member's Check- 
list, SEA Scoring Template, SEA Scoring Guide, SEA School Voting and Consensus Rating Form, SEA Planning 
Form, Appendix A, and Appendix B. These are described below. 


SEA Self-Study Guide Checklist 


This checklist delineates in chronological order the steps of the self-study process for facilitators and 
team members. The tool assists those involved in the self-study in ensuring that all tasks are completed. 


SEA Facilitator’s Checklist 


While the SEA Self-Study Guide Checklist delineates tasks of everyone involved in the self-study process, 
this checklist reflects only the responsibilities of the facilitator throughout preparation, discussion, 
and planning for next steps. This tool assists facilitators in ensuring that all tasks are completed. 


SEA Team Member's Checklist 


While the SEA Self-Study Guide Checklist delineates the tasks of everyone involved in the self-study 
process, this checklist reflects only the responsibilities of each team member throughout preparation, 
discussion, and planning for next steps. This tool assists team members in ensuring that all tasks are 
completed. 


SEA Scoring Template 


This blank template provides an opportunity for each team member to identify one or more interven- 
tions that are appropriate and relevant to the needs of the schools in the state, determine the strength of 
the associated evidence base and the fit and feasibility of the intervention, and record this information 
prior to the start of the self-study process. The form includes fields to enter the following information: 


* abroad overall area to which the intervention pertains that could be an area identified in the SEA 
Scoring Guide, or another area altogether, 


+ the specific intervention identified by research to be considered, 
« the evidence level based on a body of collected research, 


* asummary of the collection of research reviewed which may Include the results and significance 
of the studies, and 


* additional information identified locally that pertains to the needs that schools will want to con- 
sider such as school improvement plans or student achievement data. 


* Guiding questions will facilitate a discussion among team members. Guiding questions may 
include any number of factors. Some common ones to consider include: 


- the level of satisfaction among the group with the evidence-level of the intervention, 


- the extent to which the intervention was conducted on a student population that is relevant to 
the state or district context, 


- the types of schools where the intervention might work best, and 
- the possible cost/benefit of implementation. 


A rating scale is also included in the template so that, after careful consideration, self-study team 
members can determine whether they (1) do not recommend, (2) recommend, or (3) strongly recom- 
mend an intervention. The SEA Scoring Guide (described below) may be used as an example for com- 
pleting the SEA Scoring Template. 


SEA Scoring Guide 


The SEA Scoring Guide includes already identified examples of evidence-based strategies and interven- 
tions, along with a summary of the research base, the ESSA evidence-base level, state-level informa- 
tion that may be helpful to consider, and guiding questions for discussion. The content of the SEA Scor- 
ing Guide is organized into five areas: implementing systemic change; establishing strong leadership; 
improving academic instruction; developing and retaining a high-quality staff; and creating a positive 
school climate and culture. The areas chosen for the guide were based on those identified in the Insti- 
tute of Education Sciences (IES) Practice Guide Turning Around Chronically Low-Performing Schools. A 
literature review was conducted identifying interventions associated with the areas. In addition, litera- 
ture was also reviewed pertaining to the systemic interventions previously required for use in schools 
needing improvement. The SEA Scoring Guide is not meant to be an all-inclusive or recommended 
list of school improvement interventions, but rather contains examples of interventions that 
might meet the needs of schools needing comprehensive or targeted support. 


Figure 3. Areas Associated with School Improvement 


School Improvement 


It is important that interventions selected for implementation in schools in need of improvement have a strong and 
relevant evidence base and are directly related to the issues that have been identified in a needs assessment. 
Interventions may fall into a number of broad areas pertinent to school improvement. 


Sample areas that may be targeted in school improvement 


Ate S] 28 
ie. 
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Implementing Establishing Improving Developing and Creating a 
Systemic Change Strong Academic Retaining a Positive School 
Leadership Instruction High-Quality Staff Climate and 
Culture 


While the SEA Scoring Template provides\a means for SEAs to proposerto the self-study team areas and 
strategies to recommend for use in schools needing improvement, the SEA Scoring Guide provides SEAs the 
opportunity to review a number of sample strategies in five areas important to school improvement. The 
self-study team may choose to recommend some of these sample interventions for use in LEAs. 


As the facilitator and self-study team members review the information in the scoring guide, work 
through the rating system individually, and then engage in discussion, they thoughtfully consider 
whether or not to recommend an intervention for their state. The interventions recommended may 
become a menu from which LEAs may choose based on the needs of the school. It may be that an 
evidence-level is strong for an intervention, but the state has not experienced much success using that 
specific approach. Also, team members should strongly consider what has already been done in the 
state, and the effectiveness of current strategies and interventions. It may be that an evidence-level 
may be strong for an intervention but the state has not experienced much success in using that specif- 
ic approach. Perhaps some interventions should replace others based on that experience. An annotat- 
ed bibliography of the research supporting each scoring guide area is provided in Appendix A. 


SEA Voting and Consensus Rating Form 


After the SEA Scoring Guide is completed, the facilitator guides the self-study team through a consen- 
sus rating process. The team uses the SEA Voting and Consensus Rating Form to reach agreement on 
whether the proposed intervention should be recommended as an option for schools requiring com- 
prehensive or targeted support in the ESSA state plan. The most important part of this process is the 
discussion that goes into consensus rating, The scores on the SEA Voting and Consensus Rating Form 
should reflect this facilitated discussion. 


SEA Planning Form 


This form is used to establish priorities, ideas regarding resource development for LEAs, and any an- 
ticipated challenges. The facilitator leads the discussion centered on these topics and uses the form to 
record ideas. 


Appendix A. Annotated Bibliography 


This appendix describes key references that provide additional support for each of the scoring guide 
areas. Research from each study referenced in the LEA and School Scoring Guide is summarized. 


Appendix B. Theory of Action and Sample Logic Model 


This appendix provides information pertaining to theory of action and also includes a sample logic 
model to help familiarize participants with these concepts. 


Preparing for Self-Study 


In preparation for the self-study process, leadership at SEAs recruit team members to participate. Lead- 
ership at SEAs recruit members for the self-study team. The team should be comprised of a wide range 
of individuals so as to include as much knowledge and as many skills as possible. Members typically 
include researchers, content area specialists, exceptional student education (ESE) and English learner 
(EL) specialists, those involved with professional development, and senior leadership at the SEA. The 
SEA may also wish to include representatives from LEAs such as district leaders, teachers, and princi- 
pals. The names of team members and facilitator may be recorded on the SEA Voting and Consensus 
Rating Form. 


Leadership at the SEA or the team members select a dedicated and knowledgeable facilitator such as 
the school improvement director or ESSA state plan project manager. The facilitator should have deep 
content knowledge of school improvement, be well-organized, a good listener, and be able to lead a 
discussion that encourages participation from all team members. 


Once the team is established, the following steps should be followed: 


1. The facilitator studies the materials provided to conduct the self-study process so that he/she can 
effectively guide team members through the process. The facilitator gathers all pertinent data 
and evidence pertaining to the interventions. 


2. The facilitator distributes a blank SEA Scoring Template, the SEA Scoring Guide, Appendix A, Appen- 
dix B, as well as any other relevant data or evidence to each team member, and provides a time- 
line for team members to review materials. 


3. The facilitator schedules a short meeting after team members have reviewed the documents to 
discuss any questions. 


4. The facilitator asks each member to re-read the sections of the self-study guide addressing the 
collecting and evaluating of research and the ESSA Levels of Evidence. The facilitator then re- 
quests that team members research an area pertinent to school improvement in order to iden- 
tify a specific evidence-based intervention for consideration by the team during the self-study 
process, and to complete the SEA Scoring Template. Research areas could include those addressed 
in this guide: implementing systemic change, establishing strong leadership, improving academic 
instruction, developing and retaining high-quality staff, and creating a positive school climate 
and culture. Alternatively, research could include other areas selected by the team member or 
SEA. The team can work individually or with a partner or small group to identify a broad area and 
then a more specific intervention to investigate. The team may collect research on as many inter- 
ventions as they choose. Team members may wish to share their selected intervention(s) with one 
another so there is no duplication of effort. 


5. The facilitator establishes a deadline for completion and submission of the SEA Scoring Templates 
and communicates that to the team. 


6. Each team member re-reads the sections of the self-study guide addressing the collecting and 
evaluating of research and the ESSA Levels of Evidence, reviews research, completes the SEA 
Scoring Template using the SEA Scoring Guide as an example, and returns the completed template 
to the facilitator by the established deadline. 


7. The facilitator distributes the completed templates to all team members and instructs members 
to rate these strategies and interventions according to the scale on the template and to complete 
the SEA Scoring Guide. 
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8. The facilitator informs team members of the timeline for their review and schedules a consensus 
rating process meeting. 


9. Team members review the completed SEA Scoring Templates they received from the facilitator. 
They may also rate the interventions in the SEA Scoring Guide to individually reflect their thoughts 
regarding the recommendation of any interventions after reviewing the summary of research and 
any data or evidence provided by the facilitator. A team member who does not know how to rate 
a specific area may abstain from rating it. 


Collecting Research 


To collect the research necessary to identify a range of evidence-based interventions, team members 
should search professional educational journals and websites of reputable organizations. Some data- 
bases and websites to consider include: 


What Works Clearinghouse: http://ies.ed.gov/ncee/wwc/ 
ERIC: http://www.eric.ed.gov/ 
JSTOR: http://www. jstor.org/action/showAdvancedSearch 


Google Scholar: www.google.com/scholar 
Institute of Education Sciences (IES) Resources: http://ies.ed.gov 


Blueprints for Healthy Youth Development Database: http://www.blueprintsprograms.com/ 


Results First Clearinghouse: http://www.pewtrusts.org/en/research-and-analysis/issue-briefs/2014/09/ 
results-first-clearinghouse-database 


The search process begins by identifying relevant keywords. The search should not focus on just a few 
search terms, such as “school turnaround” but should be broad so as to capture as many relevant stud- 
ies as possible. Examples of keywords include: 


School turnaround Focus school Reading intervention 
School improvement Effective schools Professional development 
Low-performing schools Randomized control trial Mathematics intervention 


Keywords can be combined to look for specific ideas, such as ‘best practices’ and ‘professional develop- 
ment’ and ‘principals’ to find ways to better train school leaders. 


In addition to searching for individual articles and studies, SEAs may find resources that combine 
multiple studies in a specific area helpful. The What Works Clearinghouse (WWC) practice guides, for 
example, synthesize a large number of studies and identify those with the most supporting evidence. 
Similarly, organizations like RAND have pulled together multiple studies to provide summaries of what 
the research has found.' 


1 http://www.rand.org/pubs/research_reports/RR1550.html 


Evaluating Research 


One of the most challenging steps for many SEAs will be evaluating the research that they collect to 
match it to the appropriate levels of evidence. This section provides some general guidance on how to 
determine the level of evidence for a study; however, a number of resources exist that can help SEAs with 
this task. One is the What Works Clearinghouse’, sponsored by the Institute for Education Sciences. The 
WWC rates research studies according to a set of standards? and provides information about the rigor of 
those studies. Because the guidance around ESSA levels of evidence refer to and utilize WWC standards, 
those standards are referenced throughout this section and readers should familiarize themselves with 
them. Another resource is the Best Evidence Encyclopedia housed at Johns Hopkins University.’ 


Additionally, there are a number of organizations that SEAs can reach out to for support in evaluating 
research. Federally funded organizations such as the Regional Comprehensive Centers and content 
centers® and the Regional Educational Laboratories® are well-suited to provide states with this kind of 
support. SEAs can partner with universities that have centers and individual faculty with expertise in 
these topics. The National Network of Education Research-Practice Partnerships can provide support to 
SEAs that want to explore these kinds of research—practice partnerships.’ 


What are the ESSA levels of evidence? 


ESSA recognizes four levels of evidence. This section is designed to help SEA and LEA staff understand 
these different levels and apply them to research they are considering for school turn-around and 
related purposes. A summary of the four levels of evidence is shown in Figure 4: 


Figure 4. ESSA Levels of Evidence 


Category One 


based on at least 1 well -designed and 
well-implemented experimental study 
Demonstrates a 
statistically significant % 
effect on improving based on at least 1 well-designed and well- 
UGenEOULCOMeSs or implemented quasi-experimental study 
other relevant 
outcomes 
based on at least 1 well-designed and well- 
implemented correlational study with 
statistical controls for selection bias 


Demonstrates a 
rationale based on 
high-quality research 
findings or positive 
evaluation that such includes ongoing efforts to examine the 


activity, strategy, or effects of such activity, strategy, or intervention 
intervention is likely 


to improve student 
outcomes or other 


relevant outcomes 


Source: Source: Adapted from Chiefs for Change, 2016. 


http://ies.ed.qov/ncee/wwc/default.aspx 
http://ies.ed.qov/ncee/wwc/DocumentSum.aspx?sid=19 
http://www.bestevidence.ora/ 
http://www2.ed.gov/about/contacts/gen/othersites/compcenters.html 
http://ies.ed.gov/ncee/edlabs/ 

http://nnerpp.rice.edu/ 
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For each of the first three levels, the research studies must demonstrate a “statistically significant effect 
on improving student outcomes or other relevant outcomes.’ Statistically significant means that the 
difference observed in the study is not likely due to chance. Implied by this requirement is that the 
results are positive and not overridden by statistically significant negative results from other studies 
with moderate or strong levels of evidence. In many cases, multiple studies of the same intervention 
will yield different results and it is possible that some could be positive and others negative while all 
still being statistically significant. 


A result can be statistically significant but not substantively important. That is, a positive effect can be 
statistically significant but the effect may be so small as to be unimportant in practical terms. The im- 
pact is often described as an effect size, which is the magnitude of the difference between intervention 
groups measured as the proportion of a standard deviation. For example, an effect size of 0.25 means 
that an average student in one intervention group would be expected to have scored 0.25 standard 
deviation more had they participated in the other intervention group. The WWC considers an effect 
size of greater than or equal to 0.25 to be a substantively important difference. While not specifically 
required under ESSA, it is strongly recommended that when reviewing research the effect size should 
be considered along with the statistical significance.® 


In addition, the first three evidence levels each expect that the studies have large and multi-site 
samples and that the samples reflect populations or settings similar to those proposed to receive the 
intervention. These are critical considerations. A well-designed study with strong evidence for an in- 
tervention for early grade students may not be suitable for adolescents. Similarly, an intervention from 
a study conducted in an urban school may not be appropriate for a rural school. Ensuring that the 
sample was large, from multiple sites, and similar to the target population will increase the chances of 
success. 


Finally, the fourth level, demonstrates a rationale, can be thought of as an evidence-building oppor- 
tunity. That is, evaluation of an intervention with minimal evidence but strong supporting logic for its 
potential to improve outcomes is an opportunity to begin developing evidence of its effectiveness. 


What is strong evidence? 


Strong evidence is defined as “a well-designed and well-implemented experimental study.’ The De- 
partment of Education considers a study to be well-designed and implemented if it is meets WWC 
standards without reservations. One of the first steps in reviewing any research is to check the WWC to 
see if a study has been rated. 


But if a study has not been reviewed by the WWC, it is still possible to determine the appropriate level 
of evidence. For strong evidence this will require some form of an experiment or a regression disconti- 
nuity design. 


8 Throughout this report a number of terms are used, such as statistically significant, substantively important and intervention. A good 
resource that defines many of these terms can be found at the What Works Clearinghouse which provides an online glossary at: http://ies. 


ed.gov/ncee/wwc/Glossary.aspx. 


The essential components of an experiment in educational research include: 
- some kind of intervention or treatment designed to change outcomes, 
+ subjects who receive the intervention (typically called an experimental or treatment group), 
+ subjects who do not receive the intervention (typically called the control group), and 
+ random assignment of experimental and control groups. 


To qualify as an experiment, there must be some factor that is manipulated. This is called the treatment 
and could be a curriculum, a teaching strategy, a school policy, or anything similar. For example, a 
district might implement a new math intervention. This would be provided to some students at some 
schools but not to others. Thus, an educational aspect is changed for some individuals and held con- 
stant for others. 


The students (or teachers or schools) that receive the intervention or are part of the factor that is ma- 
nipulated are the experimental or treatment group (and possibly a comparison group). Those for whom 
instruction is unchanged are part of the control group, often called the “business-as-usual group.’ 


Note, however, that random assignment is particularly critical. Whenever two different groups receive 
different treatments, changes in outcomes could be a result of the different treatment but also because 
of differences in the groups. For example, if a school wanted to test a new reading program it might de- 
cide to give some classrooms the new program but other classrooms use the original reading program. 
This creates two groups to compare but if the students in the classes are different (maybe one group is 
more advanced than the other), differences in outcomes might be due to differences in the students and 
not the new program. The best way to overcome this risk is to randomly assign students (or teachers or 
schools) to either the treatment or control group. True random assignment helps ensure that the two 
groups are likely to be similar to each other and that any differences in outcomes are due to the treat- 
ment and not to differences between the subjects in the two groups. 


Whether or not an experiment is well-designed and well-executed is not simple to determine. There 
are numerous factors that could weaken confidence in an experiment's results, more than can be 
described here. Readers should look at resources such as the What Works Clearinghouse, which has 
developed standards to help judge the level of rigor for many educational studies. 


For this guide there are two critical limitations to focus on that can help identify studies that were not well 
designed or well executed. The first limitation is attrition. Attrition is the loss of subjects from the experiment. 
Even if the subjects are randomly assigned at the beginning, if enough members of either group leave the 
experiment, it can effectively undo the randomization process. The individuals who leave are likely to differ 
from those who stay, and, thus, if enough leave the results could be biased. The WWC provides guidance on 
appropriate levels of attrition.’ 


The second limitation is any kind of confound. A confound occurs when some aspect of the experi- 
ment is completely aligned with one aspect of the study conditions, even if all subjects were random- 
ly assigned. A confound can be thought of as an “extra” factor that was not taken into account that 
could explain the observed differences between the two groups. The most common confound occurs 
when there is only one unit (that is, teacher, classroom, school, or district) assigned to each group. For 
example, consider two classrooms taught by different teachers. One classroom comprises the inter- 
vention group and the other comprises the control group. The teachers could be randomly assigned 
to the treatment or control conditions but there would still be a confound because there was only one 
teacher in each condition. If the study found that the intervention classroom performed better than 
the control classroom, an alternative explanation for the observed difference could be related to differ- 


9 http://ies.ed.qov/ncee/wwc/Docs/referenceresources/wwc_brief_attrition 080715.pdf 
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ences between the classroom teachers and not the intervention. Another example of a confound in an 
RCT is overalignment of the outcome measure and the intervention. If the outcome measure is a direct 
measure of the intervention, then the results are confounded. An intervention that teaches specific 
spelling words and then measures the results with a test of those same words would be overaligned. 
Inclusion of a norm-referenced spelling test would be necessary to prove the intervention’s effective- 
ness beyond a taught spelling list. 


Like an experimental design, a regression discontinuity design (RDD) can meet WWC standards 
without reservations and can be considered strong evidence. An RDD determines causal impacts by 
examining interventions that occur just above and below a cut-off of some kind. In these cases, the 
cut-off, such as a cut-score on a test, splits the population of interest into two groups that can be com- 
pared. The logic is that subjects just above and just below the cut-off are likely very similar and so can 
be compared. An RDD study must meet several requirements to qualify as strong evidence, including 
establishing the equivalence between the two groups and avoiding confounds. For more information 
on how an RDD can meet WWC standards without reservations, please see the WWC reference re- 
sources. '° 


Summary of key things to look for: 
* meeting WWC standards without reservations, 
* experimental or treatment group (and the possible addition of a comparison group), 
* control group that does not receive the treatment, 
* groups formed by random assignment or a discontinuity such as a cut-score, 
¢ lowattrition, and 


« the absence of a confound. 


What is moderate evidence? 


Moderate evidence is based on at least one study using a quasi-experimental design (QED)."' What 

is the difference between an experiment and a quasi-experiment? The major difference is that a QED 
lacks random assignment of subjects to groups and instead, a QED leverages some natural change, 
such as implementation of a new program, to create treatment and control groups. QED studies are 
common because many educational policies and practices are implemented across the board or with 
a small pilot group that was not randomly assigned. For example, a few school principals might vol- 
unteer their schools to participate in a new initiative. Results from those schools might then be com- 
pared to schools that did not volunteer. This creates a treatment and a contro! group but lacks random 
assignment. As noted above, when subjects are not randomly assigned it increases the risk that any 
observed differences in outcomes are due to other factors. In this example one might wonder if the 
principals who volunteered were especially excited or interested in the intervention, or perhaps more 
creative leaders, and that it was their leadership and interest that drove changes in outcomes. 


A common QED is to compare changes in the pre-test and post-test scores for students in two differ- 
ent groups. This looks like an experiment except that the two groups were not randomly assigned. The 
researchers would try to select groups that are similar on key criteria, such as English learner status 

or economic status, so that the groups can be compared. A related approach is to statistically match 
students. One way this is done is by taking each student who received an intervention and finding a 


10 http://ies.ed.gov/ncee/wwc/Document/258 
11 Note that an RDD is a type of quasi-experimental design but it can still meet WWC standards without reservations and thus potentially 


can qualify as strong evidence. 


statistical “twin” who did not receive the intervention and then comparing results. 


As with experiments, deciding whether or not a QED is well-designed and well-executed is not simple 
to determine. Again, readers should look at resources such as the What Works Clearinghouse, which 
provides information about the level of rigor for many educational studies. A study that meets WWC 
standards with reservations qualifies as moderate evidence. 


Perhaps the single most critical factor to consider in a QED is whether or not the study was able to 
establish baseline equivalence between the two groups. As noted above, experiments use random 
assignment to try to ensure that the two groups studied are as equal as possible and often include 
pretest scores as covariates so as to improve analytic precision. Without random assignment, research- 
ers use other ways to ensure that groups are similar, such as comparing them on key variables like 
race, economic status, and test scores. Verifying that two groups are comparable on pre-test scores is 
an excellent way to establish baseline equivalence. 


Without randomized assignment there will remain a concern about unobservable group differences 
that weaken confidence in the results. For example, two students with the same pre-test scores could 
have very different levels of motivation, which could in turn result in one improving more than anoth- 
er. Concerns about unobserved differences are why even a well-executed QED is rated as only having 
moderate evidence. 


Summary of key things to look for: 
* experimental or treatment group (and the possible addition of a comparison group), 
* control group, 
* establishing or failing to establish baseline equivalence, and 


* no random assignment. 
What is promising evidence? 


Promising evidence comes from correlational studies. In a correlational study there is no assignment to 
treatment and control groups. Instead, a correlational study uses predictors or independent variables 
to look for a relationship between some factor and the outcome of interest within a group or groups 
of subjects. For example, suppose a school enacted a program to encourage students to read more 
books during the school year by offering prizes. At the end of the year a researcher might see if the 
number of books read is a good predictor of changes in student test scores. All students would be in 
the analysis so there is just one study group. The number of books serves as the independent variable 
or predictor of interest while other factors such as prior test scores might be used as control variables 
or covariates. Nonetheless, a positive association between number of books read and increase in 
student test scores would be difficult to interpret because of the lack of a control group and potential 
confounds. 


The phrase “statistical controls for selection bias” refers to some of these control variables or covariates. 
Selection bias refers to the possibility that the process of selecting or identifying the study subjects 
introduces some kind of systematic error that could invalidate the results. A common problem is selec- 
tive participation in a treatment. For example, as part of a new policy the district assigns reading coaches 
to specific schools. Because the schools were not randomly assigned, or assigned based on a cut-score, 
there would not be a good comparison group. Instead, a researcher wanting to understand if the new 
policy was effective would have to use statistical controls to try to adjust for differences between the 
schools with coaches and those without. Thus, conclusions from the available data would be limited. 


Researchers often try to overcome selection bias by checking that key factors, such as test scores and 
demographics, are similar between those receiving the treatment and those that did not. Putting 
these variables into a model allows researchers to statistically control for those factors. To meet the 
standard of promising evidence, a correlational study must have those kinds of statistical controls. 
Note that statistical controls may also be used in a QED or even an RCT to add analytic precision and to 
guard against possible confounds such as variation in district policy implementation. 


Correlational studies are considered promising evidence because there is no way to assign causality to 
the results. Mathematically, all a correlation can demonstrate is that two variables are related to each 
other. Logic might indicate a causal path, such as reading coaches lead to higher tests scores. But with- 
out random assignment there are other competing explanations for the correlation. In this example, 
reading coaches might lead to improved scores. But it is also plausible that the schools with coaches 
adopted other changes that led to higher test scores. A correlational analysis can only show an associ- 
ation, it cannot explain a causal relationship. That is why such studies are only rated promising. 


Key things to look for to identify a correlational study: 


* only one study group (no separate treatment and control groups), 


wu 


* terms such as “relationship, “covariate,” and “predictor,’ and 


* presence of statistical controls. 


What qualifies as demonstrates a rationale? 


The final level of evidence provides flexibility to work with interventions that have not been studied 
much or at all. Part of the goal for this flexibility is that allowing schools and districts to test new inter- 
ventions may add to our knowledge of what works. Note that ESSA limits the use of funds for practices 
in this category. For example, the 7% of Title |, Part A funds set aside for school improvement efforts 
must use interventions supported by research in the top three tiers. 


For the purposes of this guide, two aspects are notable. First, there should be a theory of change pro- 
viding a basis for expecting an intervention to result in an improvement. The theory of change should 
be well-constructed and well-established, such as by using a logic model. Readers are encouraged to 
develop logic models for these kinds of interventions.'? An example of a logic model for evaluating the 
effects of professional development of student reading outcomes is provided In Appendix B. 


Second, it is expected that SEAs and LEAs will carefully monitor progress of the selected strategies. 
Ideally the interventions should be evaluated through well-designed experiments but an LEA or SEA 
should at least set up an evaluation before applying the intervention. This would require, minimally, 
identifying the expected outcomes, tracking implementation, collecting follow up data, and con- 
ducting the analyses. Implementing an intervention with no way to measure or understand its conse- 
quences deprives the larger educational field an opportunity to learn more about the intervention. 


Keys to consider: 
- What is the logic model explaining the theory of change? 
+ How will the practice be evaluated? 


+ How will you know if it worked, or didn’t work? 


12 A good resource to help with creating logic models can be found at hitp://ies.ed.gov/pubsearch/pubsinfo.asp?pubid=REL2015057. 
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Follow up, Monitoring, and Evaluation 


Inherent to implementing evidence-based interventions to improve school performance is frequent 
monitoring of progress. Leaders at all levels need to address the crucially important questions of: “Are 
we on track?” and, if not, “Why not, and what are we going to do about it?” 


Most SEAs will use their existing accountability systems for monitoring and evaluation; however, there 
are ways to enhance these systems. SEAs may want to consider the following questions: 


How do SEAs and LEAs identify indicators appropriate to the interventions selected? 
How large of an improvement and how quickly should SEAs and LEAs see a difference in the indicators? 


What steps might SEAs and LEAs take to increase the intensity of the new interventions to accelerate 
student growth? 


At what point do the indicators suggest that interventions be changed? 


Answers to these questions entail that leaders build an infrastructure at the school level that addresses 
improvement in instruction, leadership, teaching, and professional development, with the necessary 
resources and accountability to be successful. 


SEA Self-Study Guide Checklist 


Self-Study Guide Checklist - Preparation 


Recruit team members which could include researchers, content area specialists, exceptional 
Task student education and English learner specialists, senior leadership, and representatives from 
LEAs such as district administrators, teachers and principals. 


Person Responsible Due Date Date Completed 


State Education Agency 
Leadership 


Follow-up Notes/Tasks 


Task Choose a knowledgeable facilitator such as a School Improvement Director or ESSA state plan 
a5 project manager. 


Person Responsible Due Date Date Completed 


State Education Agency 
Leadership or Team 


Follow-up Notes/Tasks 


Task Review materials for self-study process and gather all pertinent data and evidence pertaining 
as to the interventions. 


Person Responsible Due Date Date Completed 
Facilitator 


Follow-up Notes/Tasks 


T-1 


Distribute a blank SEA Scoring Template, SEA Scoring Guide, Appendix A, Appendix B, as well 
Task as any other relevant data or evidence to each team member. Provide a timeline for team 
members to review the materials. 


Person Responsible Due Date Date Completed 
Facilitator 


Follow-up Notes/Tasks 


Task Review all materials received from the facilitator. 
Person Responsible Due Date Date Completed 
Team Members 


Follow-up Notes/Tasks 


Task Conduct a short meeting after team members have reviewed the documents to discuss any 
questions. 
Person Responsible Due Date Date Completed 
Facilitator 


Follow-up Notes/Tasks 


T-2 


Task Attend team meeting and ask any questions to be sure the process is clear. 


Person Responsible Due Date Date Completed 


Team Members 


Follow-up Notes/Tasks 


Ask each member to re-read the sections of the self-study guide addressing the collecting 
and evaluating of research and the ESSA Levels of Evidence. Request team members to review 

Task research pertinent to an area related to school improvement to identify an evidence-based 
intervention for consideration by the self-study team. Instruct team members to complete 
the SEA Scoring Template for the intervention selected. 


Person Responsible Due Date Date Completed 


Facilitator 


Follow-up Notes/Tasks 


Task Establish a deadline for completion and submission of the SEA Scoring Templates and 
: communicate that to the team. 


Person Responsible Due Date Date Completed 


Facilitator 


Follow-up Notes/Tasks 


T-3 


Re-read the sections of the self-study guide addressing the collecting and evaluating of 
research and the ESSA Levels of Evidence. Conduct a review of research to identify a school 

Task improvement intervention to be considered for recommendation by the team. Complete the 
SEA Scoring Template, using the SEA Scoring Guide as an example, and submit the completed 
template to the facilitator by the established deadline. 


Person Responsible Due Date Date Completed 


Team Members 


Follow-up Notes/Tasks 


Task Distribute the completed SEA Scoring Templates to all team members and ask them to rate the 
as interventions according to the scale on the template and to complete the SEA Scoring Guide. 


Person Responsible Due Date Date Completed 


Facilitator 


Follow-up Notes/Tasks 


Rate the interventions on the SEA Scoring Templates (received from the facilitator) according 
Task to the rating on the template. Complete the SFA Scoring Guide after reviewing the research 
as and information provided for each intervention. Use the guiding questions to help make 
decisions. 


Person Responsible Due Date Date Completed 


Team Members 


Follow-up Notes/Tasks 


T-4 


Self-Study Guide Checklist - Discussion 
Task Conduct the first team vote in an effort to reach consensus on the ratings. 
Person Responsible Due Date Date Completed 
Facilitator 


Follow-up Notes/Tasks 


Task Guide the team discussion regarding the first vote including the rationale for decisions of 
as team members. 


Person Responsible Due Date Date Completed 
Facilitator 


Follow-up Notes/Tasks 


Participate in the discussion regarding first vote. Reconsider the first rating based on 
Task discussion. 


Person Responsible Due Date Date Completed 
Team Members 


Follow-up Notes/Tasks 


T-5 


Task Facilitate second team vote if consensus is not reached initially. 
Person Responsible Due Date Date Completed 
Facilitator 


Follow-up Notes/Tasks 


Task Participate in second team vote If consensus is not reached initially. 
Person Responsible Due Date Date Completed 
Team Members 


Follow-up Notes/Tasks 


Task Guide any discussion and records results of voting, any team thoughts, comments or concerns, 
as on the SEA Voting and Consensus Rating Form. 

Person Responsible Due Date Date Completed 
Facilitator 


Follow-up Notes/Tasks 


T-6 


Task Participate in additional discussion of voting results. 
Person Responsible Due Date Date Completed 
Team Members 


Follow-up Notes/Tasks 


Self-Study Guide Checklist - Planning 


Task Lead discussion regarding priorities, resources, and anticipated challenges and records 
as thoughts of the team on the SEA Planning Form. 


Person Responsible Due Date Date Completed 
Facilitator 


Follow-up Notes/Tasks 


Task Participate in discussion regarding priorities, resources, and anticipated challenges. 
Person Responsible Due Date Date Completed 
Team Members 


Follow-up Notes/Tasks 


T-7 


Task Mark calendar to complete tasks by established deadlines. 
Person Responsible Due Date Date Completed 
Facilitator 


Follow-up Notes/Tasks 


Record any assigned responsibilities and mark calendar to complete tasks by established 
Task deadlines. 


Person Responsible Due Date Date Completed 
Team Members 


Follow-up Notes/Tasks 


Task Schedule future meetings to assess progress. 


Person Responsible Due Date Date Completed 
Facilitator 


Follow-up Notes/Tasks 


T-8 


T-9 


SEA Facilitator’s Checklist 


Facilitator’s Checklist - Preparation 


Review materials for self-study process and gather all pertinent data and evidence 
pertaining to the strategies and interventions. 


Due Date Date Completed 


Follow-up Notes/Tasks 


Distribute a blank SEA Scoring Template, SEA Scoring Guide, Appendix A, and 
Appendix B, as well as any other relevant data or evidence to each team member. 
Provide a timeline for team members to review the materials. 


Due Date Date Completed 


Follow-up Notes/Tasks 


Conduct a short meeting after team members have reviewed the documents to 
discuss any questions. 


Due Date Date Completed 


Follow-up Notes/Tasks 


T-10 


Ask each member to re-read the sections of the self-study guide addressing 

the collecting and evaluating of research and the ESSA Levels of Evidence. 

Request team members to review research pertinent to an area related to school 
improvement to identify a specific evidence-based intervention for consideration by 
the self-study team. Instruct team members to complete the SEA Scoring Template 
for the intervention selected. 


Due Date Date Completed 


Follow-up Notes/Tasks 


Establish a deadline for completion and submission of the SEA Scoring Templates 
and communicate that to the team. 


Due Date Date Completed 


Follow-up Notes/Tasks 


Distribute the completed SEA Scoring Templates to all team members and ask them 
to rate the strategies and interventions according to the scale on the template and 
to complete the SEA Scoring Guide. 


Due Date Date Completed 


Follow-up Notes/Tasks 


T-11 


Facilitator’s Checklist - Discussion 


Conduct the first team vote in an effort to reach consensus on the ratings. 


Due Date Date Completed 


Follow-up Notes/Tasks 


Guide the team discussion regarding first vote including the rationale for decisions 
of team members. 


Due Date Date Completed 


Follow-up Notes/Tasks 


Facilitate second team vote if consensus is not reached initially. 


Due Date Date Completed 


Follow-up Notes/Tasks 


T-12 


Facilitator's Checklist - Planning 


Lead team discussion regarding priorities, resources, and anticipated challenges 
and record thoughts of the team on the SEA Planning Form. 


Due Date Date Completed 


Follow-up Notes/Tasks 


Mark calendar to complete tasks by established deadlines. 


Due Date Date Completed 


Follow-up Notes/Tasks 


T-13 


T-14 


SEA Team Member's Checklist 


Team Member's Checklist - Preparation 


Review all materials received from the facilitator. 


Due Date Date Completed 


Follow-up Notes/Tasks 


Attend team meeting and ask any questions to be sure the process is clear. 


Due Date Date Completed 


Follow-up Notes/Tasks 


Re-read the sections of the self-study guide addressing the collecting and 
evaluating of research and the ESSA Levels of Evidence. Conduct a review of 
research to identify a school improvement intervention to be considered for 
recommendation by the team. Complete the SFA Scoring Template, using the SEA 
Scoring Guide as an example, and submit the completed template to the facilitator 
by the established deadline. 


Due Date Date Completed 


Follow-up Notes/Tasks 


T-15 


Rate the strategies and interventions on the completed SFA Scoring Templates 
(received from the facilitator) according to the rating on the template. Complete the 
SEA Scoring Guide after reviewing the research and information provided for each 
intervention. Use the guiding questions to help make decisions. 


Due Date Date Completed 


Follow-up Notes/Tasks 


Team Member's Checklist - Discussion 


Participate in the discussion regarding first vote. Reconsider the first rating based 
on discussion. 


Due Date Date Completed 


Follow-up Notes/Tasks 


Participate in second team vote if consensus is not reached initially. 


Due Date Date Completed 


Follow-up Notes/Tasks 


T-16 


Team Member's Checklist - Planning 


Participate in discussion regarding priorities, resources, and anticipated challenges. 


Due Date Date Completed 


Follow-up Notes/Tasks 


Record any assigned responsibilities and mark calendar to complete tasks by 
established deadlines. 


Due Date Date Completed 


Follow-up Notes/Tasks 


T-17 


T-18 


SEA Scoring Template 


Area (choose an area from the SEA Scoring Guide, or select your own): 


Select the rating that reflects whether or not you feel this option should be included in the 
menu for selection by comprehensive or targeted support schools. 


Not recommended 
Recommended 


Strongly 
recommended 


T-19 


SEA Scoring Guide 


The areas chosen for the SEA Scoring Guide were based on those identified in the Institute of Edu- 
cation Sciences (IES) Practice Guide Turning Around Chronically Low-Performing Schools. A literature 
review was conducted identifying interventions associated with the areas. In addition, literature was 
also reviewed pertaining to the systemic interventions previously required for use in schools needing 
improvement. The SEA Scoring Guide is not meant to be an all-inclusive or recommended list of 
school improvement interventions, but rather contains examples of interventions identified in 
the practice guide that might meet the needs of schools requiring comprehensive or targeted 
support. A brief heading appears before the description of each intervention that corresponds to the 
SEA Voting and Consensus Rating Form to help team members recall the gist of each intervention as 
they complete the rating form. 


Area 1: Implementing Systemic Change 


LEAs or schools select and implement a systemic intervention which affects the organizational 
structure of the school. 


Select the rating that reflects whether or not you feel this option should be included in the 
menu for selection by comprehensive or targeted support schools. 


Reconstitution Select the Rating: 


LEAs or schools will implement a reconstitution model which Not recommended 
will replace the principal, rehire no more than 50 percent 

of the staff, and grant the principal sufficient operational Recommended 
flexibility (including staffing, calendars, schedules, and 
budgeting) to implement fully a comprehensive approach 


ee Strongl 
that substantially improves student outcomes. gy 


recommended 


Evidence Level: 

Moderate 

Summary of Research: 

One quasi-experimental study’? found improved student achievement in the first year of the re- 
form but smaller impacts in subsequent years. Over time, it does not seem that the positive impact 
on student achievement is sustained; however, it may be due to the withdrawal of support such as 
professional development that occurred in the years following the reconstitution. 


Additional Information Regarding Relevance and Appropriateness: 


Student achievement data; school improvement plans for comprehensive and targeted support 
schools; student data from schools that have reconstituted in the past. 
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Guiding Questions: 


+ Are we satisfied with the evidence level of this intervention? 

- Will this intervention meet the needs of any schools needing improvement in our state? 
- Where has a reconstitution model been implemented effectively? 

- Under what conditions were these schools successful or not? 


+ How can we help LEAs or schools ensure that the new principal and staff can make effective 
change? 


- Howdo we help LEAs or schools recruit and retain high-quality teachers? 
- How can we help LEAs or schools ensure that any initial benefit is sustained? 
+ For what schools might this be a relevant and appropriate choice? 
* Canor should this intervention be used in conjunction with other interventions? 
- What is the cost/benefit of utilizing this intervention? 
Selected Citations: 
13Strunk, K. O., Marsh, J. A., Hashim, A. K., & Bush-Mecenas, S. (2016). Innovation and a Return to the 


Status Quo A Mixed-Methods Study of School Reconstitution. Educational Evaluation and 
Policy Analysis, DOI: 0162373716642517. 


Select the rating that reflects whether or not you feel this option should be included in the 
menu for selection by comprehensive or targeted support schools. 


Transformation Select the Rating: 


LEAs or schools will implement a transformational model, Not recommended 
which by definition replaces the principal, and addresses 

various aspects at the school such as professional Recommended 
development, instructional reform, teacher evaluation and 
rewards systems, extended learning time, and community 


: Strongl 
involvement. ongly 


recommended 


Evidence Level: 


Moderate 
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Summary of Research: 


A meta-analysis of research'* was conducted regarding a transformational model as well as the 
effects associated with specific comprehensive school reform model components. Overall, the 
effects appear to be positive, especially in the instances where the intervention was in place for 
five years or more. If using an outside provider, it is important to consider the provider that is most 
appropriate for the needs of the school. While the intent was for the intervention to emphasize 
eleven specific components as identified by the U.S. Department of Education in 2002 in a com- 
prehensive manner'®, some externally developed programs emphasized some components more 
than others. 


Additional Information Regarding Relevance and Appropriateness: 
Student achievement data; school improvement plans for comprehensive and targeted support schools. 
Guiding Questions: 


- Are we satisfied with the evidence level of this intervention? 

- Will this intervention meet the needs of any schools needing improvement in our state? 
- Where has a transformational model been implemented effectively? 

- Under what conditions were these schools successful or not? 


«If LEAs or schools wish to use an outside provider to assist them, how can we help them in 
the selection process? 


- What guidance can be provided to districts if they seek to develop this model? 

+ What can we do to help promote sustainability? 

* For what schools might this be a relevant and appropriate choice? 

* Canor should this intervention be used in conjunction with other interventions? 


- What is the cost/benefit of utilizing this intervention? 
Selected Citations: 


“Borman, G. D., Hewes, G. M., Overman, L. T., & Brown, S. (2003). Comprehensive school reform and 
achievement: A meta-analysis. Review of educational research, 73(2), 125-230. 


'5May, H., & Supovitz, J. A. (2006). Capturing the cumulative effects of school reform: An 11-year 
study of the impacts of America’s Choice on student achievement. Educational Evaluation 
and Policy Analysis, 28(3), 231-257. 
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Select the rating that reflects whether or not you feel this option should be included in the 
menu for selection by comprehensive or targeted support schools. 


Transfer Control Select the Rating: 


LEAs or schools will implement a restart model which involves Not recommended 
transferring control of a school to an operator, such as a 

charter school, that has been selected through a rigorous Recommended 
review process. 


Strongly 
recommended 


Evidence Level: 
Promising 
Summary of Research: 


Only a few schools that received School Improvement Grant funds have chosen to restart by 
transferring control to a charter school. Case studies'® suggest that the autonomy associated with 
charters can be an advantage in implementing processes that may positively impact student 
achievement. That said, the restart model has had mixed results reflecting that simply converting 
a low-performing school to a charter school does not in and of itself positively impact student 
achievement”. 


Additional Information Regarding Relevance and Appropriateness: 
Student achievement data; school improvement plans for comprehensive and targeted support schools. 
Guiding Questions: 


- Are we satisfied with the evidence level of this intervention? 

* Will this intervention meet the needs of any schools needing improvement in our state? 

- Where has a restart model been implemented effectively? 

+ Under what conditions were these schools successful or not? 

« How can we ensure that the LEA/school selects the operator that best meets their needs? 
+ What review process occurs for operators? 

« For what schools might this be a relevant and appropriate choice? 

* Canor should this intervention be used in conjunction with other interventions? 


- What is the cost/benefit of utilizing this intervention? 
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Selected Citations: 


‘Corbett, J. (2015). Chartering Turnaround: Leveraging Public Charter School Autonomy to Ad- 
dress Failure. National Alliance for Public Charter Schools. 


“Herman, R. (2012). Scaling school turnaround. Journal of Education for Students Placed at Risk 
(UJESPAR), 17(1-2), 25-33. 


Select the rating that reflects whether or not you feel this option should be included in the 
menu for selection by comprehensive or targeted support schools. 


Magnet Select the Rating: 


LEAs or schools will convert to a thematic magnet school O 1 Not recommended 
resulting in a change in faculty as well as a change in student 


population. O 2 Recommended 


O 3 ~=©Strongly 


recommended 


Evidence Level: 

Moderate 

Summary of Research: 

Many years of research'® substantiates the fact that schools in need of the most improvement are 
most often schools with higher populations of minority students and students in poverty. Stud- 
ies'®?°?122 show that if the school implements a magnet program attracting students in higher 
socio-economic backgrounds, student achievement tends to increase. 


Additional Information Regarding Relevance and Appropriateness: 


Student achievement data; school improvement plans for comprehensive and targeted support 
schools; data regarding schools that have implemented magnet programs. 
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Guiding Questions: 


- Are we satisfied with the evidence level of this intervention? 

- Will this intervention meet the needs of any schools needing improvement in our state? 
- Where has the implementation of a magnet program been implemented successfully? 

- What types of magnet programs have been most successful? 

+ Under what conditions were these schools successful or not? 

+ For what schools might this be a relevant and appropriate choice? 

- Canor should this intervention be used in conjunction with other interventions? 


- What is the cost/benefit of utilizing this intervention? 
Selected Citations: 


'8Blank, R. K., Dentler, R., Baltzell, D. C., Chabotar, K (1983). Survey of magnet schools. Analyzing a 
model for quality integrated education. Final Report of a National Study 10-11 (U.S. Dept. of 
Ed.). 


Bifulco, R., Cobb, C. D., Bell, C. (2008). Do magnet schools outperform traditional public schools and 
reduce the achievement gap? The case of Connecticut's interdistrict magnet school pro- 
gram. Occasional Paper No. 167. New York: National Center for the Study of Privatization in 
Education. 


2°Gamoran, A. (1996). Student achievement in public magnet, public comprehensive, and private 
city high schools. Educational Evaluation and Policy Analysis 18, 1-18. 


*1Kahlenberg, R. D. (2009). Turnaround schools that work: Moving beyond separate but equal. Century 
Foundation. 


22Poppell, J. and Hague, S. (2001). Examining indicators to assess the overall effectiveness of 
magnet schools: A study of magnet schools in Jacksonville, Florida. Paper presented at the 
American Educational Research Association, Seattle, Washington, 10-14. 
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Area 2: Establishing Strong Leadership 


LEAs or schools will identify and employ strong leadership that can effect change quickly. 


Select the rating that reflects whether or not you feel this option should be included in the 
menu for selection by comprehensive and targeted support schools. 


Principal Commitment Select the Rating: 


LEAs or schools will ensure that the principal has a clear Not recommended 
commitment to dramatic changes from the status quo and 

can communicate the magnitude and urgency of those Recommended 
changes. 


Strongly 
recommended 


Evidence Level: 
Promising 
Summary of Research: 


It is important that principals “demonstrate commitment to developing a learning community for 
students and staff with the primary focus of the school on learning with staff and students work- 
ing together toward that goal”? School leaders also signal change through clear communication, 
creating high expectations, sharing leadership and authority, demonstrating a willingness to 
make the same types of changes asked of their staff, identifying advocates with the staff, building 
a consensus that permeates the staff, ensuring that the maximum amount of classroom time is 
focused on instruction and establishing a cohesive culture. The current principal may be able to 
signal change; however, there may need to be a change in leadership to communicate the need 
for a dramatic change in the school. 


Additional Information Regarding Relevance and Appropriateness: 


Student achievement data; school improvement plans for comprehensive and targeted support 
schools; hiring protocols from districts; school climate survey results. 
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Guiding Questions: 


« Are we satisfied with the evidence level of this intervention? 

- Will this intervention meet the needs of any schools needing improvement in our state? 

- How often are principals retained versus new principals hired? 

« How does the success of a retained principal compare to that of a newly hired principal? 

+ Under what conditions were schools that implemented this intervention successful or not? 


+ How can we ensure the principal will implement change and exhibit behaviors that impact 
student achievement? 


- What guidance can we provide LEAs and schools as they consider the retention of the current 
principal or recruitment of another? 


+ For what schools might this be a relevant and appropriate choice? 
* Canor should this intervention be used in conjunction with other interventions? 


- What is the cost/benefit of utilizing this intervention? 
Selected Citations: 


*3Herman, R., Dawson, P., Dee, T., Greene, J., Maynard, R., Redding, S., and Darwin, M. (2008). Turning 
Around Chronically Low-Performing Schools: A practice guide (NCEE #2008-4020). Washing- 
ton, DC: National Center for Education Evaluation and Regional Assistance, Institute of 
Education Sciences, U.S. Department of Education. Retrieved from http://ies.ed.gov/ncee/ 


wwe/publications/practicequides pg. 10. 


Select the rating that reflects whether or not you feel this option should be included in the 
menu for selection by comprehensive and targeted support schools. 


Principal Behaviors Select the Rating: 


LEAs or schools will ensure that principals implement O 1 Not recommended 


evidence-based behaviors shown to increase student 
achievement such as monitoring and providing feedback @ 2 Recommended 
to teachers and students, protection of instructional time, 

promoting school learning climate, supporting teachers in O 3 
professional development, emphasizing data-driven decision- 

making and positively interacting with students and teachers. 


Strongly 
recommended 


Evidence Level: 


Varies by specific behavior 
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Summary of Research: 


There are some principal responsibilities that affect student achievement more than others. There 
is evidence” that behaviors related to instructional management and internal relations impact 
student achievement while behaviors associated with organizational management and adminis- 
trative duties do not appear to impact student achievement significantly, if at all. 


Additional Information Regarding Relevance and Appropriateness: 


Student achievement data; school improvement plans for comprehensive and targeted support 
schools; principal evaluation protocol for districts; school climate survey results. 


Guiding Questions: 


- Are we satisfied with the evidence level of this intervention? 
* Will this intervention meet the needs of any schools needing improvement in our state? 


- Are there characteristics, such as years of experience, which indicate a principal would be 
more likely to exhibit these behaviors? 


* What will LEAs do to ensure that principals are engaging in behaviors that most impact stu- 
dent achievement? 


+ Under what conditions were the schools implementing this intervention successful or not? 
«+ How do we support LEAs/schools as they implement this intervention? 

* For what schools might this be a relevant and appropriate choice? 

- Canor should this intervention be used in conjunction with other interventions? 


- What is the cost/benefit of utilizing this intervention? 
Selected Citations: 


*4Osborne-Lampkin, L. T., Folsom, J. S., & Herrington, C. (2015). A systematic review of the relation- 
ships between principal characteristics and student achievement (REL 2016-091). Washing- 
ton, DC: U.S. Department of Education, Institute of Education Sciences, National Center for 
Education Evaluation and Regional Assistance, Regional Educational Laboratory South- 


east. Retrieved from http://ies.ed.gov/ncee/edlabs. 
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Select the rating that reflects whether or not you feel this option should be included in the 
menu for selection by comprehensive and targeted support schools. 


Distributed Leadership Select the Rating: 


LEAs or schools will implement a distributed leadership O 1 Not recommended 
model, transformational leadership model, or an integrated 
model to increase student achievement. O 2 Recommended 


O 3 = Strongly 


recommended 


Evidence Level: 
Promising 
Summary of Research: 


Distributed leadership and transformational leadership models positively impact student achieve- 
ment; however, it appears that the effect is indirect. These leadership styles had a significant effect 
on changes in school academic capacity, which in turn had significant effects on growth in English 
language arts and mathematics outcomes.” Studies”°”” have found that over time that schools 

with a higher level of integrated leadership (transformational and distributed) had higher academ- 
ic achievement than schools with a lower level of integrated leadership (Heck and Hallinger, 2009). 


Additional Information Regarding Relevance and Appropriateness: 


Student achievement data; school improvement plans for comprehensive and targeted support 
schools. 


Guiding Questions: 


- Are we satisfied with the evidence level of this intervention? 
- Will this intervention meet the needs of any schools needing improvement in our state? 


« Has a distributed, transformational, or integrated leadership model been implemented in 
comprehensive and targeted support schools in our state? 


« Under what conditions were these schools successful or not? 


+ What can we do to provide guidance and technical assistance to LEAs and schools to help 
them implement these leadership models? 


+ For what schools might this be a relevant and appropriate choice? 
- Canor should this intervention be used in conjunction with other interventions? 


« What is the cost/benefit of utilizing this intervention? 
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Selected Citations: 


5Louis, K. S., Leithwood, K., Wahlstrom, K. L., Anderson, S. E., Michlin, M., & Mascall, B. (2010). 
Learning from leadership: Investigating the links to improved student learning. Center for 
Applied Research and Educational Improvement/University of Minnesota and Ontario Institute 


for Studies in Education/University of Toronto, 42, 50. 


6Heck, R. H., & Hallinger, P. (2009). Assessing the contribution of distributed leadership to school 
improvement and growth in math achievement. American Educational Research Journal, 


46(3), 659-689. 


*7Osborne-Lampkin, L. T., Folsom, J. S., & Herrington, C. (2015). A systematic review of the relation- 
ships between principal characteristics and student achievement (REL 2016-091). Washing- 
ton, DC: U.S. Department of Education, Institute of Education Sciences, National Center for 
Education Evaluation and Regional Assistance, Regional Educational Laboratory South- 


east. Retrieved from http://ies.ed.gov/ncee/edlabs. 


Select the rating that reflects whether or not you feel this option should be included in the 
menu for selection by comprehensive and targeted support schools. 


Turnaround Program 


LEAs or schools provide a program such as the School 
Turnaround Specialist Program which includes substantial 
professional development to help school leaders improve 
culture, team building, data analysis, instruction and 

other aspects of the school to positively impact student 
achievement. Follow-up occurs over the course of one to two 
years. 


Evidence Level: 
Moderate 


Summary of Research: 


Select the Rating: 
Not recommended 
Recommended 


Strongly 
recommended 


A quasi-experimental four-year study”® was conducted involving schools in Cleveland and Cin- 
cinnati, Ohio. The study found statistically significant effects during and after implementing the 
School Turnaround Specialist Program and underscored the importance of strong leadership. The 
intervention entailed an intense two-year embedded professional development program in which 
leaders were given support in establishing goals, using data to make decisions regarding student 
performance, and motivating teachers. Significant growth occurred in a relatively short period of 
time. This improvement began during the two-year program and continued two years beyond. 
The analysis of data excluded schools receiving School Improvement Grants (SIG) during the time 
of the study. Although improvement was noted, the schools still fell short of the average state 


level of proficiency. 


T-30 


Additional Information Regarding Relevance and Appropriateness: 


Student achievement data; school improvement plans for comprehensive and targeted support 
schools; data or information from institutions that provide school turnaround specialist programs; 
school climate survey results. 


Guiding Questions: 


- Are we satisfied with the evidence level of this intervention? 
- Will this intervention meet the needs of any schools needing improvement in our state? 


+ Where has a school turnaround specialist program been implemented in comprehensive and 
targeted support schools in our state effectively? 


« Under what conditions were these schools successful or not? 


«+ What institutions or entities provide School Turnaround Specialists Programs or similar pro- 
grams to schools in our state? 


+ How can we ensure the program is implemented in a manner similar to the successful pro- 
gram? 


* Canor should this intervention be used in conjunction with other interventions? 
+ For what schools might this be a relevant and appropriate choice? 


- What is the cost/benefit of utilizing this intervention? 
Selected Citations: 


®Player, D., & Katz, V. (2016). Assessing School Turnaround: Evidence from Ohio. The Elementary 
School Journal, 116(4), 675-698. 
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Area 3: Improving Academic Instruction 


LEAs or schools will implement evidence-based curriculum aligned with state standards and 
assessments and use data to set goals and drive instruction for all students. 


Select the rating that reflects whether or not you feel this option should be included in the 
menu for selection by comprehensive and targeted support schools. 


Review Curricula Select the Rating: 


LEAs or schools will evaluate current curricula and Not recommended 
interventions to ensure they are evidence-based and aligned 
with state standards and assessments. Recommended 


Strongly 
recommended 


Evidence Level: 

Varies, depending on curricula 

Summary of Research: 

Research” reflects that student performance improved if instructional materials were aligned with 
state standards and assessments. The What Works Clearinghouse provides a list of many reviewed 
curricula and interventions along with their research base that are shown to improve the academ- 
ic skills of students. LEAs should incorporate consideration of the research supporting curricula 

in their review process and whenever feasible give priority to adopting curricula with stronger 
research support. 


Additional Information Regarding Relevance and Appropriateness: 


Student achievement data; instructional materials rubrics; adoption or selection process protocol; 
school improvement plans for comprehensive and targeted support schools. 
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Guiding Questions: 


- Are we satisfied with the evidence level of the curricula? 

- Will this intervention meet the needs of any schools needing improvement in our state? 
* What curricula and materials are successful schools using? 

« Under what conditions were schools implementing this intervention successful or not? 


« Are there curriculum materials or interventions used in the state that have demonstrated 
success in comprehensive and targeted support schools? 


+ What tools can be provided to help LEAs and schools evaluate curricula? 
+ For what schools might this be a relevant and appropriate choice? 
* Canor should this intervention be used in conjunction with other interventions? 


- What is the cost/benefit of utilizing this intervention? 
Selected Citations: 


Herman, R., Dawson, P., Dee, T., Greene, J., Maynard, R., Redding, S., and Darwin, M. (2008). Turning 
Around Chronically Low-Performing Schools: A practice guide (NCEE #2008-4020). Washing- 
ton, DC: National Center for Education Evaluation and Regional Assistance, Institute of 
Education Sciences, U.S. Department of Education. Retrieved from http://ies.ed.gov/ncee/ 


wwe/publications/practicequides. 


Select the rating that reflects whether or not you feel this option should be included in the 
menu for selection by comprehensive and targeted support schools. 


Analyze Data Select the Rating: 


LEAs or schools will analyze a range of data from the Not recommended 
prior year at the school level to focus on areas that need 

improvement schoolwide, at the classroom level to focus Recommended 

on teacher's instructional strengths and weaknesses, and at 
the student level to focus on the instructional needs of ALL 


students. Strongly 


recommended 


Evidence Level: 


Promising 
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Summary of Research: 


Research*°?'*? suggests that data should be analyzed at the school, classroom, and student level 
in order to identify areas of strengths and weaknesses and to determine how best to improve the 
quality of instruction. This data should not be limited to student achievement data*’, but could 
also include data reflecting the school’s climate, community, implementation of curriculum, and 
quality of instruction. In addition, it is important that the appropriate data is collected and ana- 
lyzed. Formative assessments selected for implementation must align with the standards, curricu- 
lum and the state assessment. Data should be widely distributed and teachers and administrators 
should be taught how to correctly interpret and use data so as to develop expertise in the use of 
data. 


Additional Information Regarding Relevance and Appropriateness: 


Student achievement data; school improvement plans for comprehensive and targeted support 
schools; school climate survey results. 


Guiding Questions: 


- Are we satisfied with the evidence level of this intervention? 
- Will this intervention meet the needs of any schools needing improvement in our state? 
«+ How can we ensure that appropriate data are collected and analyzed? 


- How can we ensure that data analysis occurs before the school year starts so that students 
may receive instruction that meets their needs at the beginning of the school year? 


+ What support can we provide LEAs and schools in interpreting data correctly? 
+ Under what conditions were the schools implementing this intervention successful or not? 
- How can we ensure that all subgroups are considered? 


« How can we support districts in utilizing non-academic data such as data pertaining to atten- 
dance, discipline, course enrollment and pass rates, and fiscal expenditures? 


«+ For what schools might this be a relevant or appropriate choice? 
- Canor should this intervention be used in conjunction with other interventions? 


- What is the cost/benefit of utilizing this intervention? 
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Selected Citations: 


3°Herman, R., Dawson, P., Dee, T., Greene, J., Maynard, R., Redding, S., and Darwin, M. (2008). Turning 
Around Chronically Low-Performing Schools: A practice guide (NCEE #2008-4020). Washing- 
ton, DC: National Center for Education Evaluation and Regional Assistance, Institute of 
Education Sciences, U.S. Department of Education. Retrieved from http://ies.ed.gov/ncee/ 


wwce/publications/practiceguides. 


31Anderson, S., Leithwood, K., & Strauss, T. (2010). Leading data use in schools: Organizational con- 
ditions and practices at the school and district levels. Leadership and Policy in Schools, 9(3), 
292-327. 


van Geel, M., Keuning, T., Visscher, A. J., & Fox, J. P. (2016). Assessing the Effects of a School-Wide 
Data-Based Decision-Making Intervention on Student Achievement Growth in Primary 
Schools. American Educational Research Journal, DOI: 000283 1216637346. 


33Hamilton, L., Halverson, R., Jackson, S., Mandinach, E., Supovitz, J., & Wayman, J. (2009). Using 
student achievement data to support instructional decision making (NCEE 2009-4067). Wash- 
ington, DC: National Center for Education Evaluation and Regional Assistance, Institute of 
Education Sciences, U.S. Department of Education. Retrieved from http://ies.ed.gov/ncee/ 


wwec/publications/practicequides. 


Select the rating that reflects whether or not you feel this option should be included in the 
menu for selection by comprehensive and targeted support schools. 


Progress Monitoring Select the Rating: 


LEAs or schools will progress monitor students throughout O 1 Not recommended 
the school year, analyze data, and modify instruction to meet 
the ongoing instructional needs of students. OQ 2 Recommended 


@ 3 = Strongly 


recommended 


Evidence Level: 


Moderate 
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Summary of Research: 


Teachers can use this data to determine the progress of students toward grade level standards and 
to adjust instruction accordingly.* Data should analyzed and interpreted so that teachers can de- 
velop a hypothesis regarding student learning and modify instruction to test that hypothesis and 
improve student achievement. A study*° was conducted of a computerized curriculum-based 
instructional management system implemented as an enhancement to ongoing mathematics 
instruction which enabled teachers to use data to modify instruction for students. This was shown 
to lead to an increase in student achievement in mathematics. In addition, research’’ reflects that 
a computer-adaptive literacy assessment can help to identify students at risk of not meeting grade 
level standards as well as those who are not at risk so that teachers can provide instruction accord- 
ingly. Finally, computer-adaptive assessments may be especially valuable in helping teachers to 
monitor the progress of English learners and students with learning disabilities, enabling them to 
target instruction to their needs.*® 


Additional Information Regarding Relevance and Appropriateness: 


Student achievement data and school improvement plans for comprehensive and targeted sup- 
port schools. 


Guiding Questions: 


- Are we satisfied with the evidence level of this intervention? 

- Will this intervention meet the needs of any schools needing improvement in our state? 

« Will we require specific tools for progress monitoring? 

+ Under what conditions were the schools implementing this intervention successful or not? 
- Howcan we support LEAs and schools in collecting data and analyzing it correctly? 


- How can we ensure that progress monitoring data drives continued modification of instruc- 
tion for all students in all subgroups? 


* For what schools might this be a relevant and appropriate choice? 
* Canor should this intervention be used in conjunction with other interventions? 


- What is the cost/benefit of utilizing this intervention? 
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Selected Citations: 


*4Herman, R., Dawson, P., Dee, T., Greene, J., Maynard, R., Redding, S., and Darwin, M. (2008). Turning 
Around Chronically Low-Performing Schools: A practice guide (NCEE #2008-4020). Washing- 
ton, DC: National Center for Education Evaluation and Regional Assistance, Institute of 
Education Sciences, U.S. Department of Education. Retrieved from http://ies.ed.gov/ncee/ 
wwcec/publications/practicegquides. 


sHamilton, L., Halverson, R., Jackson, S., Mandinach, E., Supovitz, J., & Wayman, J. (2009). Using 
student achievement data to support instructional decision making (NCEE 2009-4067). Wash- 
ington, DC: National Center for Education Evaluation and Regional Assistance, Institute of 
Education Sciences, U.S. Department of Education. Retrieved from http://ies.ed.gov/ncee/ 
wwc/publications/practicequides. 


°Ysseldyke, J., Spicuzza, R., Kosciolek, S., Teelucksingh, E., Boys, C., & Lemkuil, A. (2003). Using a cur- 
riculum-based instructional management system to enhance math achievement in urban 
schools. Journal of Education for Students Placed at Risk, 8(2), 247-265. 


3’7Foorman, B., Kershaw, S., Petscher, Y. (2013). Evaluating the screening accuracy of the Florida Assess- 
ments for Instruction in Reading (FAIR). (REL 2013-008). Washington, DC: U.S. Department of 
Education, Institute of Education Sciences, National Center for Education Evaluation and 
Regional Assistance, Regional Educational Laboratory Southeast. Retrieved from http://ies. 
ed.gov/ncee/edlabs/regions/southeast/pdf/REL_2013008.pdf. 


38Foorman, B., Espinosa, A., Jackson, C., Wu, Y. (2016b). Using computer-adaptive assessments of liter- 
acy to monitor the progress of English learner students. (REL 2016-149). Washington, DC: U.S. 
Department of Education, Institute of Education Sciences, National Center for Education 
Evaluation and Regional Assistance, Regional Educational Laboratory Southeast. Retrieved 
from http://ies.ed.qgov/ncee/edlabs/regions/southeast/pdf/REL_2016149.pdf. 
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Area 4: Developing and Retaining a High-Quality Staff 


LEAs or schools implement a plan for developing and retaining a high quality staff that can im- 
prove instruction and is dedicated to the school’s improvement goals. 


Select the rating that reflects whether or not you feel this option should be included in the 
menu for selection by comprehensive and targeted support schools. 


Committed Staff Select the Rating: 


LEAs or schools will build a committed staff and provide O 1 Not recommended 
professional development for teachers to improve the quality 
of instruction in the classroom and positively impact student O 2 Recommended 


achievement. 


O 3 ~=Strongly 


recommended 


Evidence Level: 
Strong 
Summary of Research: 


A common characteristic of schools that have successfully turned around is that school leaders 
chose teachers who were committed to improving the school and were qualified to implement 
high-quality instruction.*? Professional development can also help these teachers continue to im- 
prove their instruction. Nine studies*° that met the What Works Clearinghouse evidence standards, 
five of which were randomized control trials that met evidence standards without reservations, 
were examined to ascertain the effectiveness of professional development as it relates to student 
achievement. These studies focused on elementary school teachers and students and included 
four studies pertaining to reading and language arts, two related to mathematics, one focused on 
science and two on language arts, mathematics, and science. All nine studies found that teacher 
professional development had a moderate effect on student achievement. Effective professional 
development is focused on content and extends and intensifies teacher knowledge in a particular 
subject area and how students learn that content.*! A variety of approaches to professional devel- 
opment can be implemented to impact to student achievement, including the establishment of 
professional grade level teams wherein teachers can collaborate and receive professional develop- 
ment.42434 


Additional Information Regarding Relevance and Appropriateness: 


School achievement data; school improvement plans. 
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Guiding Questions: 


- Are we satisfied with the evidence level of this intervention? 
- Will this intervention meet the needs of any schools needing improvement in our state? 


- What can be done to support LEAs and schools in analyzing data to target their professional 
development plans? 


« Under what conditions were schools implementing this intervention successful or not? 


« What support can be provided for LEAs and schools as they develop their professional devel- 
opment plan? 


+ How can it be ensured that professional development plans are driven by instructional goals? 


«+ What can be done to support LEAs and schools so that they deliver high-quality professional 
development? 


* What can be done to ensure follow-up so that professional development strategies are imple- 
mented in the classroom? 


* For what schools might this be a relevant or appropriate choice? 
* Canor should this intervention be used in conjunction with other interventions? 


- What is the cost/benefit of utilizing this intervention? 
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Selected Citations: 


3°Herman, R., Dawson, P., Dee, T., Greene, J., Maynard, R., Redding, S., and Darwin, M. (2008). Turning 
Around Chronically Low-Performing Schools: A practice guide (NCEE #2008-4020). Washing- 
ton, DC: National Center for Education Evaluation and Regional Assistance, Institute of 
Education Sciences, U.S. Department of Education. Retrieved from http://ies.ed.gov/ncee/ 


wwe/publications/practicequides. 


4°Yoon, K. S., Duncan, T., Lee, S. W. Y., Scarloss, B., & Shapley, K. L. (2007). Reviewing the Evidence on 
How Teacher Professional Development Affects Student Achievement. Issues & Answers. 
REL 2007-No. 033.Regional Educational Laboratory Southwest (NJ1). 


“Early, D. M., Berg, J. K., Alicea, S., Si, Y., Aber, J. L., Ryan, R. M., & Deci, E. L. (2016). The Impact 
of Every Classroom, Every Day on High School Student Achievement: Results From a 
School-Randomized Trial. Journal of Research on Educational Effectiveness, 9(1), 3-29. 


“Antoniou, P., & Kyriakides, L. (2011). The impact of a dynamic approach to professional develop- 
ment on teacher instruction and student learning: Results from an experimental study. 
School Effectiveness and School Improvement, 22(3), 291-311. 


“Saunders, W. M., Goldenberg, C. N., & Gallimore, R. (2009). Increasing achievement by focusing 
grade-level teams on improving classroom learning: A prospective, quasi-experimental 
study of Title | schools. American Educational Research Journal, 46(4), 1006-1033. 


“4yvan Kuijk, M. F., Deunk, M. |., Bosker, R. J., & Ritzema, E. S. (2016). Goals, data use, and instruction: 
the effect of a teacher professional development program on reading achievement. School 
Effectiveness and School Improvement, 27(2), 135-156. 


Select the rating that reflects whether or not you feel this option should be included in the 
menu for selection by comprehensive and targeted support schools. 


Coaches Select the Rating: 


LEAs or schools will provide well-trained instructional coaches Not recommended 
to deliver embedded professional development for teachers 


based on data. Recommended 


Strongly 
recommended 


Evidence Level: 


Moderate 
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Summary of Research: 


The hiring of an instructional coach to provide embedded professional development can positive- 
ly impact student achievement*°”’ if the coach is well-trained and engages in behaviors such as 
modeling lessons, providing feedback, and engaging in discussions centered on data. 


Additional Information Regarding Relevance and Appropriateness: 


Student achievement data; school improvement plans for comprehensive and targeted support 
schools; data regarding the numbers and districts that have implemented instructional coaches. 


Guiding Questions: 


- Are we satisfied with the evidence level of this intervention? 
- Will this intervention meet the needs of any schools needing improvement in our state? 


* Have coaches serving in comprehensive and targeted support schools benefited student 
achievement? 


+ Under what conditions were these schools successful or not? 

«+ Should there be specific requirements for instructional coaches? 

« How can we support districts as they select coaches and train them? 

«+ How can we ensure that roles of coaches include those that benefit student achievement? 
* For what schools might this be a relevant or appropriate choice? 

* Canor should this intervention be used in conjunction with other interventions? 


- What is the cost/benefit of utilizing this intervention? 
Selected Citations: 


“Lockwood, J. R., Jennifer Sloan McCombs, and Julie Marsh. “Linking reading coaches and student 
achievement evidence from Florida middle schools.” Educational Evaluation and Policy 
Analysis 32.3 (2010): 372-388. 


46Marsh, J. A., McCombs, J. S., & Martorell, P. (2010). How Instructional Coaches Support Data-Driv- 
en Decision Making. Educational Policy, 20(10), 1-37. 


“7Matsumura, L. C., Garnier, H. E., & Spybrook, J. (2013). Literacy coaching to improve student read- 
ing achievement: A multi-level mediation model. Learning and Instruction, 25, 35-48. 
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Select the rating that reflects whether or not you feel this option should be included in the 
menu for selection by comprehensive and targeted support schools. 


Career Continuum Select the Rating: 


LEAs or schools will implement a career continuum Not recommended 
for teachers encouraging professional growth and the 

opportunity to take on leadership roles. They will compensate Recommended 
teachers based on student achievement results and their roles 


designated by the career continuum. Strongly 


recommended 


Evidence Level: 
Moderate 
Summary of Research: 


Comprehensive school reforms focused on teacher recruiting and developing high quality teach- 
ers can positively impact® student achievement. Implementing an aggressive recruitment plan in- 
cluding substantial advertising is important so that high-quality teachers are attracted to schools 
in need of improvement. In addition, establishing a career continuum can help develop and retain 
teachers by, (a) enabling teachers to assume increasing responsibilities, roles, and authority; (b) 
providing opportunities for teachers to conduct professional development in their schools; and (c) 
holding teachers accountable. Implementing a continuum and compensating teachers according 
to student achievement and their progress on the continuum yielded significant improvement 

in student achievement data compared to like schools that did not implement a comprehensive 
method of recruiting, developing, and retaining teachers. In addition, teachers working in a more 
supportive professional environment improve their effectiveness more over time than teachers 
working in less supportive contexts. 


Additional Information Regarding Relevance and Appropriateness: 


School achievement data; school improvement plans. 
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Guiding Questions: 


- Are we satisfied with the evidence level of the intervention? 
- Will this intervention meet the needs of any schools needing improvement in our state? 


- Are there districts that have established such a continuum for teachers in our state and how 
successful has that been? 


+ Under what conditions were these schools successful or not? 
+ What responsibilities or roles could be included in a career continuum? 
«+ How can we support LEAs and districts as they develop a career continuum? 
* For what schools might this be a relevant or appropriate choice? 
- Canor should this intervention be used in conjunction with other interventions? 
- What is the cost/benefit of utilizing this intervention? 
Selected Citations: 
“8Schacter, J., & Thum, Y. M. (2005). Tapping into high quality teachers: Preliminary results from the 


Teacher Advancement Program comprehensive school reform. School Effectiveness and 
School Improvement, 16(3), 327-353. 


Area 5: Creating a Positive School Climate and Culture 


LEAs or schools implement a plan to establish a positive school culture and climate that embrac- 
es high academic expectations. 


Select the rating that reflects whether or not you feel this option should be included in the 
menu for selection by comprehensive and targeted support schools. 


Safety and Community Select the Rating: 


LEAs or schools will prioritize safety, community, and Not recommended 
collaboration amongst all stakeholders including faculty, 
parents and caregivers, and the community. Recommended 


Strongly 
recommended 


Evidence Level: 


Promising 
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Summary of Research: 


Academic achievement seems to be impacted***” by a school climate and culture that addresses 
not only academic needs, but also fosters students’ feelings of safety, addresses health and mental 
health issues, and establishes high expectations for academic success. It is important to develop 
strong partnerships with parents and families, businesses, faith-based organizations, and youth 
development agencies to address these priorities beyond the school day. In addition, teacher 
effectiveness tends to improve more over time when teachers are working in supportive profes- 
sional environments as opposed to when they are working in less supportive contexts.>' 


Additional Information Regarding Relevance and Appropriateness: 
School achievement data; school improvement plans; school climate survey results. 
Guiding Questions: 


- Are we satisfied with the evidence level of this intervention? 
- Will this intervention meet the needs of any schools needing improvement in our state? 


* What districts or schools have successfully changed the culture and how did that affect stu- 
dent achievement? 


« Under what conditions were these schools successful or not? 


- What can be done to support districts as they identify areas in their culture that need to be 
improved and develop a plan for doing so? 


+ What can be done to support districts as they seek to establish partnerships with outside 
entities in their community? 


* For what schools might this be a relevant or appropriate choice? 
- Canor should this intervention be used in conjunction with other interventions? 


+ What is the cost/benefit of utilizing this intervention? 
Selected Citations: 


4°Anderson-Butcher, D., lachini, A. L., Ball, A., Barke, S., & Martin, L. D. (2016). A University-School 
Partnership to Examine the Adoption and Implementation of the Ohio Community Col- 
laboration Model in One Urban School District: A Mixed-Method Case Study. Journal of 
Education for Students Placed at Risk (JESPAR), 1-15. 


°°Tichnor-Wagner, A., & Allen, D. (2016). Accountable for Care: Cultivating Caring School Communi- 
ties in Urban High Schools. Leadership and Policy in Schools, 1-42. 


"Kraft, M. A., & Papay, J. P. (2014). Can professional environments in schools promote teacher 
development? Explaining heterogeneity in returns to teaching experience. Educational 
Evaluation and Policy Analysis, 36(4), 476-500. 
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Select the rating that reflects whether or not you feel this option should be included in the 
menu for selection by comprehensive and targeted support schools. 


Visible Change Select the Rating: 


LEAs or schools will create a climate of change evidenced by Not recommended 
visible improvements early in the turnaround process. 


Recommended 


Strongly 
recommended 


Evidence Level: 
Promising 
Summary of Research: 


Successful turnaround schools commonly implement visible changes that can be easily recog- 
nized as improvements and accomplished quickly. Although the changes made depend upon the 
school, changes can oftentimes quickly occur in the areas of use of time, resources, the physical 
plant, and student discipline.°* 


Additional Information Regarding Relevance and Appropriateness: 


School achievement data; school improvement plans; school climate survey results. 


Guiding Questions: 


- Are we satisfied with the evidence level of this intervention? 
- Will this intervention meet the needs of any schools needing improvement in our state? 


+ What districts and schools instituted changes that could be accomplished quickly and was 
that successful in benefiting student achievement? 


« Under what conditions were these schools successful or not? 


- What can be done to support districts as they make decisions regarding what types of posi- 
tive changes could be made quickly? 


* For what schools might this be a relevant or appropriate choice? 
* Canor should this intervention be used in conjunction with other interventions? 


- What is the cost/benefit of utilizing this intervention? 
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°*Herman, R., Dawson, P., Dee, T., Greene, J., Maynard, R., Redding, S., and Darwin, M. (2008). Turning 
Around Chronically Low-Performing Schools: A practice guide (NCEE #2008-4020). Washing- 
ton, DC: National Center for Education Evaluation and Regional Assistance, Institute of 


Education Sciences, U.S. Department of Education. Retrieved from http://ies.ed.gov/ncee/ 
wwe/publications/practicequides. 
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SEA Voting and Consensus Rating Form 


This form is used to document the results of consensus ratings by the self-study team. The facilitator 
leads the team in consensus voting which consists of several steps: 


1. Vote. Ask each team member to provide a numerical ranking (1-3) for each of the areas. 


2. Identify frequency. |dentify the most frequent ranking (if three team members vote 3, five vote 2, 
and two vote 1, the most frequent ranking that team members votes is 2). 


3. Discuss the rationale of the high frequency ranking. Ask ateam member who selected the high 
frequency ranking to talk about what motivated that vote. 


4. Discuss the rationale of lower frequency rankings. Ask other team members to talk about why they 
voted in a particular way. 


5. Vote. Use numeric ranking to vote a second time. Team members may change their votes based 
on the discussion. 


6. Record rating on this form. \f there is a consensus (typically determined by majority vote), record 
the high frequency ranking. If consensus is not reached (there is a tie), continue discussing and 
voting until consensus is reached. 


7. Continue across areas of the self-study guide and include strategies and interventions submitted 
by team members. 


SEA Self-Study Team: 
Facilitator: 

Team Member: 

Team Member: 

Team Member: 

Team Member: 


Team Member: 
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SEA Consensus Form: 


NR = Not Recommended 
R = Recommended 
SR = Strongly Recommended 


Scoring Guide Area 


1. Implementing Systemic 
Change 


2. Establishing Strong 
Leadership 


3. Improving Academic 
Instruction 


4. Developing and Retaining a 
High Quality Staff 


Consensus Rating 


Intervention 1 
(reconstitution) 


Intervention 2 
(transformation) 


Intervention 3 
(transfer control) 


Intervention 4 
(magnet) 


Intervention 1 
(principal commitment) 


Intervention 2 
(principal behaviors) 


Intervention 3 
(distributed leadership) 


Intervention 4 
(turnaround program) 


Intervention 1 
(review curriculum) 


Intervention 2 
(analyze data) 


Intervention 3 
(progress monitoring) 


Intervention 1 
(committed staff) 


Intervention 2 
(coaches) 


Intervention 3 
(career continuum) 
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O 
O 
O 
O 
O 
O 
O 
O 
O 
O 
@ 
O 
O 
O 


O Ba O Be O BAO BO BO BS OO 


Scoring Guide Area Consensus Rating 


5. Creating a Positive School Intervention 1 
Climate and Culture (safety and community) 


Intervention 2 
(visible change) 


6. Team-proposed Area Intervention 1 


Intervention 2 


Intervention 3 


7. Team-proposed Area Intervention 1 


Intervention 2 


Intervention 3 


8. Team-proposed Area Intervention 1 


Intervention 2 


Intervention3 


9. Team-proposed Area Intervention 1 


Intervention 2 


Intervention 3 


O 
O 
O 
O 
O 
O 
O 
O 
O 
O 
O 
O 
O 
O 


QO Bae O Be O BS O BS O BS O BO 
O Be O Be O BS O BS O BS O BO 
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SEA Planning Form 


(to be completed by the facilitator) 


After the SEA Voting and Consensus Rating Form has been completed, the facilitator will lead a discus- 
sion with the team regarding priorities for action. The facilitator will then complete the planning form 
based on the thoughts of the team. While many priorities may be identified, the team may choose to 
focus on only a few at any one time so as not to be overwhelmed. The discussion may also include 
next steps for developing and disseminating resources to LEAs. Any challenges and ideas to meet 
those challenges may also be captured. 


AREA: 


1. Based on group discussion and consensus ratings, list the top priorities pertaining to the 
recommendations of interventions for school improvement. 


2. What are next steps in addressing the priorities? Consider timelines and who will be responsible. 


3. What resources need to be provided for LEAs? Consider timelines and who will be responsi- 
ble for development and dissemination. 


4. What potential challenges are anticipated? How will they be addressed? Who will be respon- 
sible for addressing these challenges? 


5. Who will be responsible for ensuring that priorities and resource development and dissemi- 
nation are occurring according to the established timeline? 
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Appendix A. Annotated Bibliography 


This appendix describes key references that provide additional support for each of the Scoring Guide 
areas. 


Scoring Guide Area 1: Implementing Systemic Change 


Strunk, K. O., Marsh, J. A., Hashim, A. K., & Bush-Mecenas, S. (2016). Innovation and a Return to the 
Status Quo A Mixed-Methods Study of School Reconstitution. Educational Evaluation and Policy 
Analysis, DOI: 0162373716642517. 


This study of a small set of schools that were reconstituted in an urban area (pg. 555) found 
that students in reconstituted schools experience sizable and significant gains in ELA during 
the first two years of reconstitution, but insignificant effects for math. Changes in the state- 
wide assessment prevented these schools from being studied in subsequent years (pg. 556); 
however, case study data reflected that while reconstitution initially improves the student 
achievement at the school, the effects diminish over time (pg. 570). The authors suggest that 
it may be helpful for districts to maintain support in the form of funding and providing other 
resources for several years (pg. 571). 


Borman, G. D., Hewes, G. M., Overman, L. T., & Brown, S. (2003). Comprehensive school reform and 
achievement: A meta-analysis. Review of educational research, 73(2), 125-230. 


The authors note that there are limitations on the overall quantity and quality of the research 
base; however, the effects of the comprehensive school reform model appear promising. 
Schools that implemented the model for five years or more showed particularly strong effects 
(pg. 125). 


May, H., & Supovitz, J. A. (2006). Capturing the cumulative effects of school reform: An 11-year study 
of the impacts of America’s Choice on student achievement. Educational Evaluation and Policy 
Analysis, 28(3), 231-257. 


The authors present the results of an 11-year longitudinal study of the America’s Choice comprehen- 
sive school reform design focused on student learning gains. The study was conducted in Rochester, 
New York and compared test scores of students attending America’s Choice schools with the scores of 
students who attended other schools and students who attended the same schools before America’s 
Choice was implemented. There were significant annual effects, which accumulated over time in the 
elementary and middle grades (pg. 231). This study also found that over time, particularly after the 
fifth year of implementation, the effects dropped off and that although the effects were significant, 
students who were working below grade level did not catch up with grade-level peers (pg. 253). The 
America’s Choice model emphasizes ongoing assessment and differentiation of instruction (pg. 252). 


Corbett, J. (2015). Chartering Turnaround: Leveraging Public Charter School Autonomy to Address 
Failure. National Alliance for Public Charter Schools. 


The authors reflect that only a few districts or schools have chosen to restart schools as char- 
ters. Case studies indicate several benefits of restarting a school as a charter including the free- 
dom to hire, place, and remove staff; provide professional development and incentive; to use 
time as deemed best for students; adopt curriculum and implement other academic services; 
allocate dollars to priority areas and to own and maintain facilities (pg. 20). Case studies reflect 
improvements in student performance in some schools (pg. 12). 


Herman, R. (2012). Scaling school turnaround. Journal of Education for Students Placed at Risk (JESPAR), 
17(1-2), 25-33. 


The author reflects that evidence regarding the effects of charter schools and education man- 
agement organizations focuses on primarily on charter schools in general. Student achieve- 
ment results are mixed when comparing student performance in charter schools to that of 
students in other schools (jg. 27). It is unclear if true flexibility is afforded to charter schools 
that are low-performing or if that flexibility matters when it comes to student achievement 
(pg. 28). 


Blank, R. K., Dentler, R., Baltzell, D. C., Chabotar, K (1983). Survey of magnet schools. Analyzing a model for 
quality integrated education. Final Report of a National Study 10-11 (U.S. Dept. of Ed.). 


The authors examine using magnet programs to improve the quality of education in urban areas 
and also to facilitate integration of schools. They note that “While desegregation does not ‘pre- 
dict’ quality, within magnets a racial balance does predict academic gains. Integration and quality 
are highly associated; each is a correlative facet of effectiveness,” (pg. 134). A variety of factors in 
success are noted for schools that were studied. These include leadership of the principal, parental 
support, coordinated instructional program, and use of community resources (jpg. 403, 412). 


Bifulco, R., Cobb, C. D., Bell, C. (2008). Do magnet schools outperform traditional public schools and re- 
duce the achievement gap? The case of Connecticut's interdistrict magnet school program. Oc- 
casional Paper No. 167. New York: National Center for the Study of Privatization in Education. 


Results of a study conducted in Connecticut's central cities indicate that “attendance at an 
interdistrict magnet high school has positive effects on the math and reading achievement 
of central city students and that interdistrict magnet middle schools have positive effects on 
reading achievement,’ (pg. 323). 


Gamoran, A. (1996). Student achievement in public magnet, public comprehensive, and private city 
high schools. Educational Evaluation and Policy Analysis 18, 1-18. 


The author reflects that results of a study in American cities indicating that magnet schools were 
more effective than public comprehensive high schools in raising proficiency in science, reading 
and social studies (pg 1). In addition, principals of magnet schools reported “slightly more posi- 
tive academic climates, on average, than principals in comprehensive schools,’ (pg. 8). 


Kahlenberg, R. D. (2009). Turnaround schools that work: Moving beyond separate but equal. Century 
Foundation. 


The author states that there are “a number of studies over the past quarter-century that have 
found that magnet schools have higher levels of achievement than do other schools, and 
produce faster achievement gains in most subjects” (pg. 8). In addition, the magnet model is 
one where “schools seek to improve the performance of low-income students by drawing into 
a high-poverty school a contingent of middle class students” (pg. 8). 


Poppell, J. and Hague, S. (2001). Examining indicators to assess the overall effectiveness of magnet 
schools: A study of magnet schools in Jacksonville, Florida. Paper presented at the American 
Educational Research Association, Seattle, Washington, 10-14. 


A study of magnet schools in Duval County Public Schools in Florida found that academic 
achievement of students attending magnet schools exceeded that of students who attended 
nonmagnet schools. The schools were established as part of a plan to desegregate the district 


(pg. 1). 
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Scoring Guide Area 2: Establishing Strong Leadership 


Herman, R., Dawson, P., Dee, T., Greene, J., Maynard, R., Redding, S., and Darwin, M. (2008). Turning 
Around Chronically Low-Performing Schools: A practice guide (NCEE #2008-4020). Washington, 
DC: National Center for Education Evaluation and Regional Assistance, Institute of Education 
Sciences, U.S. Department of Education. Retrieved from http://ies.ed.gov/ncee/wwec/publica- 


tions/practicequides. 


This practice guide addressing turnaround of chronically low-performing schools recommends 
that strong leadership signal the need for dramatic change. It is important that principals 
“demonstrate commitment to developing a learning community for students and staff with 
the primary focus of the school on learning with staff and students working together toward 
that goal” (pg. 10). School leaders also signal change through clear communication, creating 
high expectations, sharing leadership and authority, demonstrating a willingness to make 
the same types of changes asked of their staff, identifying advocates with the staff, building a 
consensus that permeates the staff, ensuring that the maximum amount of classroom time is 
focused on instruction and establishing a cohesive culture (pg. 10-11). The current principal 
may be able to signal change; however, there may need to be a change in leadership to com- 
municate the need for a dramatic change in the school (pg. 11). 


Osborne-Lampkin, L. T., Folsom, J. S., & Herrington, C. (2015). A Systematic Review of the Relationships 
between Principal Characteristics and Student Achievement. (REL 2016-091). Washington, DC: U.S. 
Department of Education, Institute of Education Sciences, National Center for Education Eval- 
uation and Regional Assistance, Regional Educational Laboratory Southeast. Retrieved from 


http://ies.ed.gov/ncee/edlabs. 


The authors “describe the literature on principal behaviors linked to improved student achieve- 
ment” (pg. 9). The behaviors are organized into five domains which include instructional 
management, internal relations, organizational management, administrative duties, and 
external relations. Under instructional management, behaviors such as “monitoring and pro- 
viding feedback to teachers and student,’"“having a vision for learning,’“providing support and 
professional development to teachers,’ and “using data to drive decision-making,’ were found 
to have positive relationships with student achievement. One study found that “promoting 
high standards for student learning (r= .55-.61) and having a rigorous curriculum (r =.42-.47) 
were most highly correlated with English language arts achievement in grades 3-5 and that 
performance accountability was significantly correlated in grade 3 (r = .37; Reardon, 2011)” (pg. 
9-10). Eight of nine studies examined found a positive relationship between internal relations 
and student achievement while three of five studies reflected positive relationships between 
the time that principals spent on organizational management and student achievement. No 
studies found any relationship between principals’ time spent on administrative duties and 
student achievement. There were mixed results when it came to time spent devoted to exter- 
nal relationships and student achievement with school-communhity links n high-poverty and 
rural schools positively related to student achievement. 


Louis, K. S., Leithwood, K., Wahlstrom, K. L., Anderson, S. E., Michlin, M., & Mascall, B. (2010). Learning 
from leadership: Investigating the links to improved student learning. Center for Applied Re- 
search and Educational Improvement/University of Minnesota and Ontario Institute for Studies in 
Education/University of Toronto, 42, 50. 


The authors of this study examined leadership at the school, district, and state level with the 
purpose to “identify the nature of successful educational leadership and to better understand 
how such leadership can improve educational practices and student learning” (pg. 7). At the 
school level, the authors reflected that among other findings that “collective leadership has a 
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stronger influence on student achievement than individual leadership” (pg. 19). Data suggests 
that “collective leadership has modest but significant indirect effects on student achievement” 
(pg. 28) as it positively effects teacher variables such as work setting and motivation which, in 
turn, impact student achievement. 


Heck, R. H., & Hallinger, P. (2009). Assessing the contribution of distributed leadership to school im- 
provement and growth in math achievement. American Educational Research Journal, 46(3), 
659-689. 


The authors of this study examined the relationship between distributed leadership and 
academic capacity when observed over time and how distributed leadership impacts school 
improvement and subsequent growth in math (pg. 677). They “found support for the hypoth- 
esis that school leadership and capacity building are mutually reinforcing in their effects on 
each other over time,’ and that “changes in these mutually reinforcing constructs were also 
positively associated with school growth rates in math. The effect size for change in academic 
capacity was almost 0.2” (pg. 679-680). 


Osborne-Lampkin, L.T., Folsom, J. S., & Herrington, C. (2015). A Systematic Review of the Relationships 
between Principal Characteristics and Student Achievement. (REL 2016-091). Washington, DC: 
U.S. Department of Education, Institute of Education Sciences, National Center for Education 
Evaluation and Regional Assistance, Regional Educational Laboratory Southeast. Retrieved 
from http://ies.ed.gov/ncee/edlabs. 


The authors examined a study investigating distributed or collaborative leadership. The study 
found that although there was no evidence of direct effect of collaborative or distributed lead- 
ership on student achievement, there was consistent indirect effects (pg. 12). The study found 
significant effect on changes in school academic capacity “which in turn had a significant 
effect on growth in student achievement in English language arts” (pg. 12). 


Player, D., & Katz, V. (2016). Assessing School Turnaround: Evidence from Ohio. The Elementary School 
Journal, 116(4), 675-698. 


The authors of this study examined “a sample of 20 Ohio schools that participated in a school 
turnaround program and found that participating schools experienced meaningful improve- 
ments in student achievement after completing the two-year program” (pg. 675). These 
schools investigated the implementation of a School Turnaround Specialist Program (STSP) 
where it was required that the principal and at least half of the school’s prior staff would be 
replaced. That said, the principal was replaced in only six of the 20 schools (pg. 691). Profes- 
sional development to the principal and other leaders of the school the summer before the 
program was implemented and considerable support was provided to the principals through 
mentoring (pg. 679). “The schools examined as a part of this study demonstrated statistically 
and practically significant growth in student achievement within 2 years of participating in 
STSP" (pg. 694). 


A-4 


Scoring Guide Area 3: Improving Academic Instruction 


Herman, R., Dawson, P., Dee, T., Greene, J., Maynard, R., Redding, S., and Darwin, M. (2008). Turning 
Around Chronically Low-Performing Schools: A practice guide (NCEE #2008-4020). Washington, 
DC: National Center for Education Evaluation and Regional Assistance, Institute of Education 
Sciences, U.S. Department of Education. Retrieved from http://ies.ed.gov/ncee/wwe/publica- 


tions/practicequides. 


The practice guide states that a “comprehensive curriculum review can ensure that the cur- 
riculum aligns with state and local standards and meets the needs of all students (pg. 19). In 
addition, the What Works Clearinghouse establishes levels of evidence for assessing the quality 
of evidence supporting educational programs and practices (pg. 3). 


The practice guide also indicates that schools need to “examine student achievement data 

to identify gaps and weaknesses in student learning....they can examine student learning 
through standards-based assessments and classroom assessments’ (pg. 17). In addition, 
“school personnel can also look at data on factors that contribute to or impeded student learn- 
ing, such as attendance, discipline, and fiscal expenditures” (pg. 17). 


Anderson, S., Leithwood, K., & Strauss, T. (2010). Leading data use in schools: Organizational conditions 
and practices at the school and district levels. Leadership and Policy in Schools, 9(3), 292-327. 


“This study examined data use and conditions influencing data use by typical principals and 
teachers, as well as the relationship between data use and student performance” (pg. 292). 
The authors note that data should be accessible, timely, and valid. In addition, the staff should 
have the expertise to analyze the data correctly (pgs. 296-297). “It is not data use per se that 
affects the quality of teaching and learning; rather it is the appropriateness of actions actually 
taken based on data-informed decisions about the nature of the problem and how it might be 
solved (pg. 321). 


van Geel, M., Keuning, T., Visscher, A. J., & Fox, J. P. (2016). Assessing the Effects of a School-Wide Da- 
ta-Based Decision-Making Intervention on Student Achievement Growth in Primary Schools. 
American Educational Research Journal, DOI: 0002831216637 346. 


This study investigated a school-wide data-based decision-making (DBDM) intervention in 
primary schools in The Netherlands. The intervention involved a two-year training course in 
DBDM for primary school teams (pg. 366). It was hypothesized that “implementing DBDM will 
lead to changes in teacher’s classroom practices, which in turn will lead to student achieve- 
ment growth in mathematics” (pg. 370). Results indicated that the intervention “can lead to a 
considerable improvement in the correct interpretation of student achievement data” (pg. 387) 
and there were positive effects on student achievement. In addition, the intervention “signifi- 
cantly improved the performances of students in low socioeconomic schools” (pgs. 360-361). 


Hamilton, L., Halverson, R., Jackson, S., Mandinach, E., Supovitz, J., & Wayman, J. (2009). Using student 
achievement data to support instructional decision making (NCEE #2009-4067). Washington, DC: 
National Center for Education Evaluation and Regional Assistance, Institute of Education Sci- 
ences, U.S. Department of Education. Retrieved from http://ies.ed.gov/ncee/wwec/publications/ 


practiceguides. 


This practice guide recommends that a variety of data is collected about student learning. 
Multiple data sources are important because, “no single assessment provides all the infor- 
mation teachers need to make informed instructional decisions” (pg. 11). Data collected may 
include “curriculum-based unit tests; class projects; classwork and homework; records from 
parent meetings and phone calls; classroom behavior charts; individualized education plans; 
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and prior data from students’ cumulative folders” (pg. 13). 


Herman, R., Dawson, P., Dee, T., Greene, J., Maynard, R., Redding, S., and Darwin, M. (2008). Turning 
Around Chronically Low-Performing Schools: A practice guide (NCEE #2008-4020). Washington, 
DC: National Center for Education Evaluation and Regional Assistance, Institute of Education 
Sciences, U.S. Department of Education. Retrieved from http://ies.ed.gov/ncee/wwec/publica- 
tions/practicequides. 


The practice guide reflects that schools in need of improvement should “monitor progress and 
make adjustments” (pg. 17). Once schools have identified areas that needed improvement 
and develop a plan to improve instruction, they should continually monitor progress. In the 
schools cited in the practice guide, all of them used benchmark assessments or in some way 
systematically monitored student achievement and progress toward instructional goals (pg. 
17). This was done so instruction could be modified as needed. 


Hamilton, L., Halverson, R., Jackson, S., Mandinach, E., Supovitz, J., & Wayman, J. (2009). Using student 
achievement data to support instructional decision making (NCEE #2009-4067). Washington, DC: 
National Center for Education Evaluation and Regional Assistance, Institute of Education Sci- 
ences, U.S. Department of Education. Retrieved from http://ies.ed.gov/ncee/wwe/publications/ 
practicequides. 


This practice guide recommends that teachers interpret data, develop a hypothesis about how 
to improve student learning (pg. 14), modify instruction to test the hypothesis, and continue 
the cycle to increase student learning (pg. 15). Modifying instruction may mean allocating 
more time, reordering the curriculum, identifying particular students in need of assistance 
with specific skills, attempting to teach complex skills in new ways, improving alignment be- 
tween performance expectations among grade levels, or better aligning curricular alignment 
in the school (pg. 15). 


Ysseldyke, J., Spicuzza, R., Kosciolek, S., Teelucksingh, E., Boys, C., & Lemkuil, A. (2003). Using a curricu- 
lum-based instructional management system to enhance math achievement in urban schools. 
Journal of Education for Students Placed at Risk, 8(2), 247-265. 


The authors reflect that in order to improve teaching and learning, systematic, usable informa- 
tion regarding individual student performance and progress at the classroom level must be 
available (pg. 247). The study examined the “use of a computerized curriculum-based instruc- 
tional management system in addition to ongoing math instruction” (pg. 248). The system al- 
lowed teachers to differentiate instruction based on data. Results reflect a positive effect with 
students in classrooms implementing the system demonstrating more growth than students 
in classrooms that did not implement the system (pg. 259). 


Foorman, B., Espinosa, A., Jackson, C., Wu, T. (2016b). Evaluating the screening accuracy of the Florida 
Assessments for Instruction in Reading (FAIR). (REL 2013-008). Washington, DC: U.S. Department 
of Education, Institute of Education Sciences, National Center for Education Evaluation and 
Regional Assistance, Regional Educational Laboratory Southeast. Retrieved from http://ies. 
ed.qov/ncee/edlabs/regions/southeast/pdf/REL_2013008.pdf. 


This study examined the association between student performance on the 2012 Florida 
Comprehensive Assessment Test (FCAT) and their scores on the Florida Assessment for Instruc- 
tion in Reading (FAIR) during three assessment periods throughout the year. In addition, the 
authors looked at the effects of adding FAIR as a means of preventing errors while identifying 
students in need of intervention (pg. i). The study showed a strong correlation between FAIR 
FCAT Success Probability (FSP) scores and performance on the 2012 FCAT at all grade levels. 

In addition, while FCAT could be used to identify students at risk/not at risk of meeting grade 
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level standards the following school year, implementing FAIR as a progress monitoring tool 
throughout the school year decreased the percentage of students that were misidentified. For 
example, “using FAIR FSP scores (which combine the FAIR Reading Comprehension Assessment 
with the 2011 FCAT 2.0 score) reduced underidentification from 21 percent in grade 4 to 4-6 
percent” (pg. 9). 


Foorman, B., Kershaw, S., Petscher, Y. (2013). Using computer-adaptive assessments of literacy to monitor 
the progress of English learner students. (REL 2016-149). Washington DC: U.S. Department of Ed- 
ucation, Institute of Education Sciences, National Center for Education Evaluation and Regional 
Assistance, Regional Education Laboratory Southeast. Retrieved from http://ies.ed.gov/ncee/ 


edlabs/regions/southeast/pdf/REL_2016149.pdf. 


This study, conducted in a large urban district in Florida, examined how teachers and school 
staff administered computer-adaptive assessments of literacy to English learner students in 
grades 3-5 and how they used the assessments to monitor students’ growth in literacy skills. 
(pgs. 1-2). “Reliably measuring the literacy skills of English learner students can be challenging. 
Assessments typically address only grade-level proficiency, do not provide instructionally rel- 
evant information, and are not developmentally scaled to measure change over time” (pg. 2). 
The Florida Assessments for Instruction in Reading (FAIR) K-2 system was used because of the 
low level of English proficiency. The study found that teachers partnered with each other so 
that the assessment could be delivered within the required timeframe. Students’ literacy skills 
improved during the course of the year, but most students remained at the same grade level 
in the FAIR K-2 system at the end of the school year. Teachers found the data helpful as they 
could use it to plan and adjust instruction as needed. 


Scoring Guide Area 4: Developing and Retaining a High-Quality Staff 


Herman, R., Dawson, P., Dee, T., Greene, J., Maynard, R., Redding, S., and Darwin, M. (2008). Turning 
Around Chronically Low-Performing Schools: A practice guide (NCEE #2008-4020). Washington, 
DC: National Center for Education Evaluation and Regional Assistance, Institute of Education 
Sciences, U.S. Department of Education. Retrieved from http://ies.ed.gov/ncee/wwec/publica- 


tions/practicequides. 


The authors reflect that “the school leader needs to build a staff that is committed to the 
school’s improvement goals and qualified to meet them” (pg. 27). In addition, while not a focus 
of the specific recommendation in the practice guide, the author's state that “professional 
development to help staff reach the school’s goals is an essential element of all school reform 
efforts and should be a part of turnaround schools,’ (pg. 27). 


Yoon, K. S., Duncan, T., Lee, S. W. Y., Scarloss, B., & Shapley, K. L. (2007). Reviewing the Evidence on How 
Teacher Professional Development Affects Student Achievement. Issues & Answers. REL 2007- 
No. 033. Regional Educational Laboratory Southwest (NJ1). 


The authors examined nine studies that addressed the effect of teacher professional develop- 
ment on student achievement in mathematics, science, and reading or English language arts. 
Five of the studies were randomized controlled trials and met the What Works Clearinghouse 
evidence standards without reservation. Four studies met the evidence standards with reser- 
vations (pg. iii). In all studies the professional development provided was directly to teachers 
and not through a “train the trainer” approach. It was delivered by those who created the 
professional development. It was also found that studies that had “more than 14 hours of pro- 
fessional development showed a positive and significant effect on student achievement from 
professional development” (pg.’3). Further, the authors state that “First, professional develop- 
ment enhances teacher knowledge and skills. Second, better knowledge and skills improve 
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classroom teaching. Third, improved teaching raises student achievement. ...If a teacher fails 
to apply new ideas from professional development to classroom instruction, students will not 
benefit from the teacher's professional development” (pg. 4). 


Early, D. M., Berg, J. K., Alicea, S., Si, Y., Aber, J. L., Ryan, R. M., & Deci, E. L. (2016). The Impact of Every 
Classroom, Every Day on High School Student Achievement: Results From a School-Random- 
ized Trial. Journal of Research on Educational Effectiveness, 9(1), 3-29. 


Professional development was a key component of the set of instructional improvement 
interventions that were examined by this study. The study was conducted in high schools and 
included professional development for both mathematics and English teachers (pg. 3). The 
authors explain that professional development should be content focused, “meaning that it 
extends and intensifies teacher knowledge of a subject area and how children learn subject 
specific content” (pg. 5-6). Students attending treatment schools had higher math scores than 
those who attended schools not in the treatment group (pg. 19). Although the professional 
development component alone was not studied, it was a major component of the interven- 
tion set. 


Antoniou, P., & Kyriakides, L. (2011). The impact of a dynamic approach to professional development 
on teacher instruction and student learning: Results from an experimental study. School Effec- 
tiveness and School Improvement, 22(3), 291-311. 


This study investigated a dynamic integrated approach to professional development as 
opposed to a holistic approach. The dynamic approach focused on factors that describe the 
teachers’ instructional role and are associated with student outcomes such as questioning, 
classroom assessment, and teacher-modeling while the holistic approach focused on teachers’ 
beliefs, experiences, and reflection on teaching practices (pgs. 291-292). The study found that 
teachers that had participated in the dynamic approach to professional development were 
more effective than those participating in the holistic approach model (pg. 303). 


Saunders, W. M., Goldenberg, C. N., & Gallimore, R. (2009). Increasing achievement by focusing 
grade-level teams on improving classroom learning: A prospective, quasi-experimental study 
of Title | schools. American Educational Research Journal, 46(4), 1006-1033. 


The authors conducted a quasi-experimental investigation focused on the effects of estab- 
lishing grade-level teams focused on student learning on student achievement. Professional 
development was provided to the principal and the teachers on establishing the teams and 
professional development occurred during team meetings. Student achievement at schools 
in the treatment group improved at a faster rate than student achievement at comparable 
schools who did not implement grade-level teams (pg. 1). 


van Kuijk, M. F., Deunk, M. |., Bosker, R. J., & Ritzema, E. S. (2016). Goals, data use, and instruction: the 
effect of a teacher professional development program on reading achievement. School Effec- 
tiveness and School Improvement, 27(2), 135-156. 


The authors of this study investigated whether student reading comprehension could be 
improved through a professional development program emphasizing goals, data use, and 
instruction (pg. 1). Second and third grade teachers received 40 hours of professional develop- 
ment over the course of the school year. They attended meetings after school and completed 
homework assignments. Participation was voluntary and free of charge; however, no addi- 
tional compensation was provided to teachers (pg. 140). The study found a positive effect on 
student achievement and at the end of the program “students in the experimental condition 
were more than half a year ahead of students in the control condition” (pg. 150). 
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Lockwood, J. R., Jennifer Sloan McCombs, and Julie Marsh. “Linking reading coaches and student 
achievement evidence from Florida middle schools.” Educational Evaluation and Policy Analysis 
32.3 (2010): 372-388. 


The authors conducted an evaluation of a statewide reading coach program in Florida middle 
schools. “Using achievement data from nearly 1,000 Florida middle schools from the 1997- 
1998 through 2005-2006 school years, we find that receiving a state-funded coach was associ- 
ated with statistically significant improvements in average annual reading achievement gains 
for two of the four cohorts of schools analyzed” (pg. 1). It is possible that the lack of effects for 
one of the cohorts (2006) may have been due to the fact that implementation had taken place 
for only one year. The other cohort (2004) was small and it is possible that idiosyncrasies of 
the schools came into play (pg. 383). Overall, “our results might be more supportive of positive 
coaching effects than the simple count of statistically significant findings would imply” (pg. 
383). 


Marsh, J. A., McCombs, J. S., & Martorell, P. (2010). How Instructional Coaches Support Data-Driven 
Decision Making. Educational Policy, 20(10), 1-37. 


The authors examined how coaches support data-driven decision-making and “the extent 

to which these efforts are associated with improvements in teaching and student achieve- 
ment” (pg. 873). Data support was one of many activities to which coaches devoted their time. 
Coaches spent time administering and coordinating assessments, working with individual 
teachers, managing resources and materials, as well as working with groups of teachers. They 
also, in some cases, devoted time to non-coaching tasks such as substitute teaching or per- 
forming “duties” such as lunch duty or bus duty. More experienced coaches spent more time 
in supporting data-driven decision-making. A positive relationship was found between data 
analysis and student achievement (pg. 898). 


Matsumura, L. C., Garnier, H. E., & Spybrook, J. (2013). Literacy coaching to improve student reading 
achievement: A multi-level mediation model. Learning and Instruction, 25, 35-48. 


The authors conducted a group-randomized trial in which schools within one district received 
a content-focused coach (CFC) and other schools continued with literacy coaching that was 
standard practice in the district (pg. 38). The CFC coaches helped teachers become more pro- 
ficient at planning, teaching, and reflecting on their lessons and emphasized the Questioning 
the Author (QtA) approach which is a discussion-based approach to reading comprehension 
(pg. 37). Coaches met with teachers in weekly grade level teams and monthly in their class- 
rooms. The study found that the CFC program had a positive effect on the quality of classroom 
discussions and “by the end of that academic year, students in the CFC schools demonstrated 
significantly higher reading achievement than their comparison group peers” (pg. 44). In addi- 
tion, the CFC program helped to close the gap between ELL and non-ELL students in the study 
(pg. 44). 


Schacter, J., & Thum, Y. M. (2005). TAPping into high quality teachers: Preliminary results from the 
Teacher Advancement Program comprehensive school reform. School Effectiveness and School 
Improvement, 16(3), 327-353. 


This study investigated whether schools implementing the Teacher Advancement Program 
(TAP) outperformed comparable schools on an annual basis, outperformed its controls, wheth- 
er fidelity to implementation influenced student achievement and teacher satisfaction with 
the program (pg. 334). “By aggressively recruiting new teachers, providing a career continuum, 
introducing teacher-led professional development, implementing rigorous teacher account- 
ability, and paying teachers based on their position, teaching skills and how much their 
students achieve, TAP schools change their organizational structure to support and reward 
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high-quality instruction” (pg. 327). The student achievement in TAP schools grew significantly 
when compared to the controls although the magnitude of the gains varied by school and 
fidelity of implementation (pg. 327). 


Scoring Guide Area 5: Creating a Positive School Climate and Culture 


Anderson, S., Leithwood, K., & Strauss, T. (2010). Leading data use in schools: Organizational conditions 
and practices at the school and district levels. Leadership and Policy in Schools, 9(3), 292-327. 


This study examined a model designed to support school improvement efforts by empha- 
sizing youth development, parent and family engagement and support, health and social 
services and community partnerships (pg. 192). The authors looked at the types of capacity-re- 
lated innovations developed to support the model, whether school-level perceptions improve 
throughout implementation, and whether or not school-level indicators of academic achieve- 
ment improve over the course of implementation. The study found that roles and responsibil- 
ities of staff changed to focus on the model and that innovations occurred that resulted in the 
use of data for planning. There was an improvement in the perception of the school climate 
and in academic motivation and implementation resulted in increased student achievement 
(pg. 198). 


Tichnor-Wagner, A., & Allen, D. (2016). Accountable for Care: Cultivating Caring School Communities in 
Urban High Schools. Leadership and Policy in Schools, 1-42. 


The authors of this study examined the caring practices in two higher performing and two 
lower performing urban high schools. It was found that “higher performing schools demon- 
strated caring communities, where interpersonal relationships and high academic expecta- 
tions were prevalent throughout the school” (pg. 406). Factors such as “strong leadership sup- 
port, caring as a core school value, and abundant curricular and extracurricular structures” (pg. 
406) were less prevalent in lower performing schools that had only isolated instances of care. 


Kraft, M. A., & Papay, J. P. (2014). Can professional environments in schools promote teacher develop- 
ment? Explaining heterogeneity in returns to teaching experience. Educational Evaluation and 
Policy Analysis, 36(4), 476-500. 


The authors examined whether a supportive professional environment is associated with 
teacher improvement over time (pg. 476). The professional environment included factors such 
as the extent to which the school was a safe environment and order prevailed, the opportunity 
for peer collaboration, the support of the principal, the opportunity for teachers to participate 
in professional development, the respect, openness, and commitment to student achievement 
and a teacher evaluation process that provided teachers with meaningful feedback which 
could be used to improve instruction (pg. 480). The study concluded that teachers “working in 
more supportive professional environments improve their effectiveness more over time than 
teachers working in less supportive environments” (pg. 476). 


Herman, R., Dawson, P., Dee, T., Greene, J., Maynard, R., Redding, S., and Darwin, M. (2008). Turning 
Around Chronically Low-Performing Schools: A practice guide (NCEE #2008-4020). Washington, 
DC: National Center for Education Evaluation and Regional Assistance, Institute of Education 
Sciences, U.S. Department of Education. Retrieved from http://ies.ed.gov/ncee/wwe/publica- 


tions/practicequides. 
The practice guide recommends providing “visible improvements early in the turnaround 
process” (pg. 22). These can include making improvements to the physical environment such 


as painting, ensuring the school building and grounds are clean, and fixing anything that is 
broken (pg. 25). In addition, establishing a safe and orderly environment by implementing an 
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approach to discipline that demonstrates the presence of administrators and safety officers, in- 
volves parents, and provides a means of dispensing discipline swiftly and fairly can also impact 
student learning and be implemented fairly quickly. 
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Appendix B. Theory of Action 
and Sample Logic Model 


It is important that a strong theory of action and a logic model be in place when choosing interven- 
tions to utilize in schools needing comprehensive or targeted support. This is particularly important 
when using studies that fall under “demonstrates a rationale” level of evidence. A theory of action may 
be described as follows: 


- Aligns intended theory with the realities of work within an actual organization. 


- Connects strategy to the actions and relationships critical to good instruction and student learn- 
ing. 


- Identifies the mutual dependencies that are required to get the complex work of...improvement 
done. 


+ Grounded in research or evidence-based practice. 


- Begins with a statement of a causal relationship between what I/we do and what constitutes a 
good result in the organization. 


- High leverage for achievement and equity. 


- Powerful enough to transform programs and practices. 


- Adapted from Instructional Rounds in Education — Elizabeth A. City, Richard F. Elmore, Sarah E. 
Fiarman and Lee Teitel, 2009 


The development of a theory of action may help educators consider the rationale behind their choice 
of interventions and convey the thinking behind the decisions they make. A general theory of action 
can be the basis for the creation of a more specific logic model. 


Logic models are helpful in planning and monitoring evaluations of interventions. They can guide 
those working with the interventions develop a clear and complete understanding of the activities in- 
volved in the intervention along with the intended outcomes. They can also help those involved in the 
implementation of the intervention to think through the details of Implementation systematically. In 
addition, a logic model may help educators formulate evaluation questions and ensure that the gen- 
eral evaluation questions are clear, specific, and actionable. An example of a logic model developed by 
the Regional Educational Laboratory Pacific is below: 
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Figure B1. Sample logic model for a teacher training program on alternative reading strategies 


Research-based 
guidance on 
reading 
strategies 


Curriculum 
coordinators 


Elementary 
school teachers 


Number and 
type of guides 
and sample 
lessons for each 
grade level 


Develop and 
provide teaching 
guides and 
sample lessons 


Conduct 
teacher 
workshops 


Number of 
participants per 
workshop and 
total hours each 
participant 
attended the 
workshop 


Increased 
teacher 
knowledge of 
multiple 
instruction 
strategies to 
teach reading 


Increased 
teacher 
knowledge of 
reading content 


Source: Kekahio, W., Cicchinelli, L., Lawton, B., & Brandon, P. R., 2014. 


Resources: 


Increased 
teacher use of 
alternative 
strategies for 
presenting 
reading content 


Increased 
positive student 
attitudes toward 

learning 


Increased 
student 
understanding of 
reading content 


Increased 
student reading 
test scores 


Institute of Education Science: http://ies.ed.gov/pubsearch/pubsinfo.asp?)pubid=REL2015057 
REL Pacific: http://relpacific.mcrel.org/resources/elm-app/ 


W.K. Kellogg Foundation: http://www.smartgivers.org/uploads/logicmodelguidepdf.pdf 
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